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Abstract 

Real  sensor  imagery  is  obtained  with  actual  sensors,  installed  on  their  platforms  and 
imaging  real  environments.  Simulated  sensor  imagery  is  used  to  create  sensor  imagery 
generated  under  simulated  environments  and  conditions.  Imagery  simulation  capabilities 
have  been  created  to  support  procedural  training,  weapon-system-training  requirements, 
mission  rehearsal,  system  research  and  development  and  human-factors  studies.  The 
present  study  was  designed  to  investigate  sensor  imagery  simulation  capabilities  and 
requirements,  with  emphasis  on  SAR  (Synthetic  Aperture  Radar)  and  FLIR  (Forward 
Looking  Infrared)  imagery  and  to  develop  a  framework  for  analyzing  required  simulation 
fidelity  for  various  applications. 

Physics-based  simulation  tools  are  most  recent  and  advanced.  These  tools  model  the 
components  that  impact  real  sensor  images:  the  environment  (terrain  and  atmosphere), 
the  radiometric  equations,  and  the  sensor  system.  A  survey  was  conducted  to  identify 
existing  sensor  imagery  simulation  products  and  applications.  The  study  also  includes  a 
literature  review  on  object  recognition  and  sensor  image  quality  attributes  that  affect 
object  recognition  and  acquisition  performance. 

The  concept  “simulation  fidelity”  is  used  to  describe  the  effective  differences  between  a 
real  and  a  simulated  system.  Ideally  the  simulated  image  should  be  identical  to  its  real 
world  coimterpart  (i.e.,  perfect  simulation  fidelity).  However,  perfect  simulations  may  not 
be  feasible  (technically  or  financially)  and  may  not  always  be  necessary.  The  central 
issue  of  the  present  study  is  to  identify  fidelity  requirements  for  various  uses  of  simulated 
sensor  imagery.  In  theory,  state-of-the-art,  physics-based  simulation  programs,  are 
capable  of  producing  very  high-fidelity  simulations  of  sensor  imagery.  In  practice, 
however,  simulation  fidelity  depends  on  the  completeness  and  accuracy  of  all 
components  of  the  simulation,  i.e.,  the  representation  of  terrain,  terrain-objects  and 
human-placed  objects,  atmospheric  conditions,  the  radiometric  equations  and  the 
representation  of  the  specific  sensor  system.  The  efforts  required  to  produce  high-fidelity 
simulations  may  be  tremendous,  therefore,  the  necessary  fidelity  of  the  simulation  should 
be  determined  by  the  requirements  of  the  application.  These  requirements  were  identified 
for  various  applications,  first  on  the  basis  of  the  theoretical  analysis  and  then  through 
structured  interviews  with  Subject  Matter  Experts  (SME). 

The  main  obstacles  for  creating  high  fidelity  FLIR  and  SAR  simulation  with  physics- 
based  simulation  tools,  for  both  SAR  and  FLIR,  are  related  to  the  very  limited  availability 
of  material-maps  and  to  the  difficulty  in  representing  human-made  ground  objects.  In 
addition,  interactions  between  components  within  and  between  objects  (particularly  in 
FLIR)  are  hard  to  represent. 

The  sensor-imagery  simulation-fidelity  model,  presented  herein,  proved  effective,  it  was 
easy  for  the  SME  to  relate  to  the  concepts  and  come  up  with  the  required  evaluations. 
SME  provided  significant  data  on  the  required  simulation  fidelity  of  various  image 
components  in  simulated  SAR  and  FLIR  images,  for  different  tasks  (photo  interpretation, 
reconnaissance,  target  recognition  &  designation)  and  for  different  simulation 
applications  (training,  mission  rehearsal). 
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Introduction 


Background 

Real  sensor  imagery  can  be  obtained  with  actual  sensors,  installed  on  their  platforms  and 
imaging  real  environments.  Simulated  sensor  imagery  is  used  to  create  sensor  imagery 
generated  under  simulated  conditions  and  environments.  Imagery  simulation  capabilities 
have,  in  general,  been  created  to  support  procedural  training,  weapon-system-training 
requirements,  operator’s  training  and  mission  rehearsal,  system  research  and  development 
and  human-factors  studies. 

No  research  has  been  conducted  to  establish  the  content  fidelity  requirements  of 
simulated  imagery  in  the  various  contexts,  and  particularly,  in  the  context  of  object 
recognition  and  target  acquisition  performance. 

Objectives 

The  present  study  was  designed  to  perform  a  survey  of  sensor  imagery  simulation 
capabilities  and  requirements,  with  emphasis  on  SAR  (Synthetic  Aperture  Radar)  and 
FLIR  (Forward  Looking  Infrared)  imagery.  The  study  includes  a  literature  search  and 
review  on  sensor  image  quality  attributes  and  their  effects  on  object  recognition  and 
acquisition  performance.  Applications  for  simulated  imagery  have  been  identified  and 
their  features  are  described  in  the  report.  Qualitative/quantitative  requirements  for  these 
applications  have  been  derived  in  terms  of  image  quality  and  image  content.  A  set  of 
criteria  for  the  performance  of  pattern  and  object  recognition  and  target  acquisition  tasks, 
with  each  type  of  simulated  imagery  (FLIR  and  SAR),  has  been  developed.  The  proposed 
set  of  criteria  for  the  evaluation  of  required  simulation  fidelity  of  various  types  and 
purposes  was  examined  with  the  help  of  ten  expert  users  of  sensor  imagery  (subject 
matter  experts  -  SME). 

Sensor  Imagery 

Vision  is  the  primary  source  of  information  about  the  environment.  The  eyes  of  humans 
and  of  most  animals  are  sensitive  to  visible  light,  which  consists  of  a  very  narrow  portion 
of  the  electromagnetic  spectrum.  As  a  result,  the  ability  to  see  depends  on  reflected  or 
emitted  light  that  reaches  the  eye.  When  there  is  no  light  (e.g.,  at  night)  or  when  the  light 
is  absorbed  or  scattered  before  it  reaches  the  eye  (e.g.,  during  fog)  or  when  an  object  is 
concealed  by  another  object  which  does  not  transmit  light  (e.g.,  camouflage),  then  the 
ability  to  see  is  degraded. 

Various  devices  have  been  developed  to  transform  information  that  is  contained  in  other 
bands  of  electromagnetic  radiation  into  visible  representations.  Well-known  examples  are 
X-rays,  radar  (radio  wavelength)  and  FLIR  (thermal)  imaging.  Some  of  these  techniques 
are  active  i.e.,  radiation  is  artificially  emitted  and  then  recorded  (e.g..  X-ray,  radar);  and 
some  are  passive  i.e.,  natural  radiation  is  captured  by  sensors  (e.g.,  FLIR  imaging,  light 
amplification,  multi-spectral). 

The  visual  images  that  are  produced  by  most  of  these  techniques  differ  distinctly  from 
regular,  visual  band  images.  In  regular  monochrome  images  (e.g.,  black  and  white 
television),  the  distribution  of  gray  shades  represents  the  relative  brightness  and 
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reflectance  of  objects  in  the  scene.  Usually,  the  resulting  image  seems  “natural”  i.e.,  it  is 
similar  to  what  the  eye  is  accustomed  to  seeing  in  everyday  life.  In  other  types  of  sensor 
imagery  the  distribution  of  gray  shades  in  the  image  represents  different  phenomena.  As  a 
result,  these  images  appear  significantly  different  from  regular  images  of  the  same  scene 
and  also,  from  each  other.  Therefore,  the  interpretation  of  each  type  requires  special 
skills.  The  special  nature  of  FLIR  and  SAR  imagery  is  discussed  in  more  detail  below. 

FLIR  Imagery 

Capturing  electromagnetic  radiation  in  the  IR  band  and  transforming  it  into  visible 
images  creates  FLIR  imagery.  Every  object,  whose  temperature  is  above  absolute 
zero,  emits  electromagnetic  radiation.  The  spectral  range  and  the  intensity  in  which 
most  radiation  is  emitted  depend  on  the  temperature  and  the  emissivity  (emission 
efficiency)  of  the  object.  The  radiant  emittance  from  the  surface  of  the  earth  is  similar 
to  the  emittance  of  a  blackbody  at  300  degrees/Kelvin.  It  is  near  zero  in  the  visible 
band  and  reaches  its  peak  around  10  microns.  This  range  is  loosely  defined  as  the  IR 
band.  Typically,  FLIR  systems  operate  either  in  the  3  -  5  micron  band,  or  in  the  8  - 
12  micron  band,  in  which  there  is  good  atmospheric  transmission.  The  background 
contrast  is  higher  in  the  8  -  12  micron  band,  resulting  in  a  more  distinct 
representation  of  image  details  (i.e.,  better  image  quality). 

Images  in  the  natural  visual  spectrum  (0.4  -  0.7  microns)  consist  of  light  that  is 
emitted  from  one  remote  source  (sun,  moon)  and  reflected  from  features  in  the 
environment.  The  brightness  of  any  given  point  in  the  environment  depends  on  the 
reflection  of  source  light  incident  on  that  point.  Most  terrestrial  materials  have  good 
reflectance  in  the  visible  band;  whereas,  only  an  insignificant  portions  of  incident 
infrared  radiation  is  reflected.  Therefore,  FLIR  images  are  produced  primarily  hy 
emitted  IR  radiation.  In  other  words,  while  regular  images  depend  on  the  immediate 
presence  of  a  light  source,  the  thermal  image  represents  the  “memory”  of  previously 
accumulated  energy  (and  self-  generated  heat),  and  does  not  depend  on  the  immediate 
presence  of  an  external  source  of  energy.  As  a  result,  thermal  images  are  capable  of 
extending  human  vision  into  the  night.  In  addition,  infrared  radiation  is  less  affected 
than  light  by  small  atmospheric  particles  and  to  some  extent,  may  penetrate  through 
smoke,  haze  and  fog.  It  may  also  penetrate  some  sorts  of  materials  and  “see”  hot 
objects  (e.g.,  engines)  through  camouflage.  FLIR  systems  are  used  as  night  vision 
devices  on  many  different  types  of  platforms  (e.g.,  fighter  aircraft,  helicopters, 
remotely  piloted  and  unmanned  air  vehicle,  guided  missiles,  tank  and  armored 
vehicles)  and  for  a  variety  of  applications  (reconnaissance,  navigation,  night  flight, 
night  driving,  target  acquisition). 

The  dynamic  temperature  range  that  IR  detectors  are  capable  of  handling  is  much  too 
large  to  be  usefully  displayed  at  once.  For  example,  the  temperature  range  between  a 
“hot”  target  (e.g.,  an  aircraft)  and  a  “cold”  background  (e.g.,  the  sky),  may  span  more 
than  300  degrees/centigrade  and  the  IR  detector  may  be  able  to  sense  temperature 
differences  of  less  than  0.5  degrees/centigrade.  Therefore,  FLIR  systems  are  provided 
with  level-control  that  sets  the  range  of  temperatures  to-be-displayed,  and  gain- 
control  that  determines  the  gray-shades  mapping  of  temperatures  within  that  range.  In 
addition,  most  FLIR  systems  can  control  polarity,  i.e.,  the  way  in  which  gray-shades 
represent  temperatures.  Polarity  may  he  set  to  “black  hot”  depicting  hot  objects  as 
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dark  gray  shades  and  cold  objects  as  bright  gray  shades  or  it  may  be  set  to  the 
opposite  “white  hot”. 

FLIR  images  differ  significantly  from  regular  images  (Brickner  and  Zvuloni,  1993). 

•  The  distribution  of  gray  shades  in  the  FLIR  image  represents  the  mapping  of 
relative  IR  emittance  of  objects  in  the  scene  (determined  by  objects’ 
temperatures  and  emissivity)  and  not  their  brightness  and  reflectance. 

•  The  IR  image  is  created  by  emitted  rather  than  reflected  light  and  therefore, 
lacks  the  systematic  shading  characteristic  of  daytime  natural  scenes. 

•  Regular  images  and  IR  images  change  over  time  in  different  ways.  Regular 
images  tend  to  change  gradually  and  systematically  (e.g.,  the  brightness  of  the 
whole  image  is  reduced  and  all  shadows  grow  longer  towards  dusk).  IR 
images  change  in  much  less  predictable  manners,  because  the  relative  heat 
emittance  of  objects  in  the  scene  may  rapidly  change  and  even  reverse  (e.g., 
metal  objets  which  absorb  and  emit  heat  rapidly,  seem  hot  during  the  day  but 
cool  off  rapidly  during  the  night).  Furthermore,  when  the  heat  emittance  of  an 
object  is  close  to  that  of  the  background  (the  “crossover”  point)  it  may  become 
invisible. 

•  Human-made  and  natural  sources  of  heat  (e.g.,  fire,  friction,  chemical 
processes),  may  create  unique  thermal  signatures  that  are  significantly  different 
from  the  regular  representation  of  the  same  objects. 

SAR  Imagery 

Synthetic  aperture  radar  (SAR)  is  a  coherent  radar  system  located  on  a  moving 
platform.  The  radar  transmits  a  narrow  pulse  of  electromagnetic  energy,  which  is 
reflected  from  the  area  on  the  ground  and  is  returned  to  a  receiver.  The  synthetic 
aperture  is  formed  as  the  physical  antenna  (installed  on  an  aircraft  or  satellite)  moves 
through  space.  The  image  formation  is  usually  based  on  the  assumption  that  only  a 
linear  motion  of  constant  velocity  is  present.  SAR  requires  motion  compensation  to 
remove  the  effects  of  other  motion.  The  aircraft  motion  is  acquired  from  the  inertial 
navigation  system.  Then  the  receiver  Local  Oscillator  (LO)  is  offset  by  an  appropriate 
frequency  to  remove  the  instantaneous  Line-of-Sight  Doppler  from  the  radar  signal 
that  is  due  to  the  aircraft  motion. 

SAR  offers  two  compelling  advantages  over  conventional  (electro-optical)  sensing 
technologies:  standoff  range  and  adverse  weather  capabilities.  SAR  images  can  be 
formed  with  effectively  no  loss  in  resolution  up  to  the  limits  of  the  system’s 
stabilization  and  motion  compensation  capabilities.  SAR  sensors  can  “see”  through 
clouds  and  through  light  rain.  Further,  depending  on  their  coverage  mode  and  data 
processing  limitations,  SAR  sensors  can  be  capable  of  high  area  coverage  rates. 

These  attributes  make  SAR  imaging  a  valuable  resource  for  tactical  and  theater 
airborne  reconnaissance,  surveillance  and  target  acquisition  applications. 

SAR,  however,  is  a  non-literal  imaging  sensor,  that  is,  the  imagery  produced  by  a 
SAR  does  not  resemble  a  photograph  taken  of  the  same  scene.  The  intensity  values  in 
the  SAR  image  are  proportional  to  the  radar  cross  sections  (RCS)  of  the 
corresponding  points  in  the  ground  scene  (and  not  to  their  visible  wavelength 
reflectance).  The  impulse  response  function  of  the  SAR  (the  fundamental 
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determinant  of  system  resolution)  includes  side  lobes.  Thus,  the  return  from  a  point 
on  the  ground  may  include  energy  contributed  by  adjacent  scatterers.  In  addition,  high 
order  acceleration  deviations  of  the  moving  platform,  from  the  computed  virtual 
antenna,  may  contribute  visual  noise. 

The  “shadows”  in  a  S  AR  image  are  caused  by  the  active  illumination  of  the  scene  by 
the  transmitting  radar  (and  not  by  the  sun  angle).  If  the  grazing  angle  is  low  and  the 
area  mountainous,  large  areas  may  be  concealed  from  radiation  and  appear  as  black 
spots.  The  perspective  of  a  SAR  image  is  that  of  an  observer  looking  dovm  on  to  the 
scene  from  directly  above,  as  the  radar  illuminates  it  from  one  side. 

SAR  images  appear  very  different  from  both  regular  and  FLIR  images. 

•  The  distribution  of  gray  shades  in  the  SAR  image  represent  the  proportional 
RCS  of  the  corresponding  points  in  the  ground  scene,  as  “viewed”  by  the  radar 
transmitter,  and  is  not  correlated  with  gray  shade  distribution  of  regular  images 
or  FLIR  images. 

•  Unlike  regular  imagery  and  FLIR  imagery,  SAR  images  are  not  affected  by 
time-of-day  and  are  only  marginally  affected  by  atmospheric  conditions. 

•  SAR  imagery  is  strongly  affected  by  the  transmission  angle  of  the  radar, 
which  changes  the  effective  radar  cross  section  of  objects  in  the  scene  and  their 
appearance  on  the  display. 

•  Until  recently,  most  operational  SAR  systems  provided  much  lower  resolution 
than  TV  and  FLIR  systems.  These  low-resolution  SAR  images  presented  very 
crude  images  of  the  world  that  could  be  interpreted  only  by  pattern  (e.g., 

SAMS  site)  or  by  reference  to  other  information  sources  (e.g.,  detecting 
changes  between  a  new  image  and  a  previously  interpreted  one).  Recently, 
high-resolution  SAR  systems  are  becoming  available.  These  systems  may 
allow  the  recognition  of  objects  with  sufficiently  large  RCS  and  distinct 
contour  lines  (e.g.,  roads,  buildings,  military  vehicles). 
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Simulation  of  Sensor  Imagery 

Real  sensor  imagery  is  acquired  and  recorded  during  real-world  missions.  Obviously,  the 
use  of  a  real  system  for  non-operational  purposes  is  rather  costly  and  limited. 
Furthermore,  the  collection  of  actual  imagery  to  meet  the  requirements  of  methodical 
training  or  of  an  experimental  design  (different  grazing  angles,  target  orientation  etc.) 
may  be  too  complicated  and  expensive  to  achieve  with  sufficiently  accurate  control  over 
the  parameters  of  interest.  Therefore,  there  is  a  strong  need  for  simulated  sensor  imagery. 

Uses  of  simulate  sensor  imagery 

Simulated  sensor  imagery  may  serve  various  different  purposes: 

Operator  training  and  mission  rehearsal 

General  (procedural)  training:  Incorporating  sensor  imagery  into  a  simulated 
environment  (e.g.,  flight  simulators)  where  sensor  imagery  is  not  the  primary  goal 
of  training  but  rather  part  of  the  operating  procedure  (e.g.,  general  pilot  training). 

Specific  training:  Training  of  sensor  imagery  operators  (e.g.,  SAR  imagery 
analysts,  FLIR  weapon  system  operators). 

Mission  plaiming  and  rehearsal:  If  a  mission  environment  can  be  simulated,  the 
operator  may  use  it  to  plan  the  mission  and  gain  effective  rehearsal  of  the  mission. 

System  research  and  development 
Sensor  system  design. 

Sensor  system  analysis. 

Sensor  performance  analysis  and  evaluation. 

Automatic  target  recognition  (ATR)  development,  research  and  training. 

Human  factors  studies 
Operator  performance  studies 

Imagery  exploitation 

Predictions  of  what  a  target  should  look  like,  to  be  used  in  comparison  with  actual 
sensor  imagery. 

Generate  and  evaluate  softcopy  imagery-interpretation-key-materials  for  use  by 
imagery  analysts. 

Types  of  simulation  systems 

Over  the  years,  various  approaches  to  sensor  imagery  simulation  have  been  developed, 
based  on  existing  technological  capabilities.  Bair  (1996)  described  the  evolution  of 
airborne  A/G  (air  to  ground)  radar  simulation  (DRLMS  -  Digital  Radar  Landmass 
simulation).  In  the  mid  1960’s  glass  plates  with  flying  spot  scanners  were  used  to 
generate  simulated  radar  displays.  Terrain  boards  were  used  in  the  1970’s.  Neither  of 
these  methods  produced  accurate  depictions  of  angle  or  resolution  effects.  Another 
difficulty  with  early  DRLMS  was  the  lack  of  adequate  source  data  from  which  to  build 
the  landmass  database.  This  problem  was  abated  in  the  early  1970’s  with  the  introduction 
of  digital  mapping  systems  that  eventually  led  to  the  modem  Digital  Terrain  Elevation 
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Data  (DTED)  and  Digital  Feature  Analysis  Data  (DFAD).  By  1980,  hardware-intensive 
implementations  of  radar  models  were  constructed.  These  were  big,  expensive  and  hard- 
to-maintain  devices.  In  1985  a  software-only  solution  was  developed,  based  on  large 
mainframe  computers.  This  led  to  the  use  of  UNIX  based  workstations  which  are  still  in 
extensive  use.  Recently,  some  PC  based  solutions  have  also  become  available  (e.g., 
RadBase). 

A  similar  but  somewhat  different  trend  can  be  identified  in  FLIR  simulations.  A  model 
board  with  heated  metal  objects  representing  targets  was  used  at  the  US  Army  Night 
Vision  Laboratory  in  the  1970’s.  Recorded  sensor  imagery  was  used  as  a  substitute  for 
real  imagery  in  many  early  studies  (e.g.,  Mochamuk,  Gaudio  and  Suwe,  1981).  Recorded 
images  were  used  to  create  ’’closed  loop”  simulations  by  recording  the  images  on  video 
disk  and  providing  a  limited  level  of  interactive  dynamics.  In  addition,  various  image 
processing  functions  could  be  used  to  manipulate  the  image,  and  simulate  gain,  level, 
polarity  etc.  (e.g.,  Brickner  and  Foyle,  1995).  Image  processing  techniques  have  been 
used  to  make  daytime  images  look  similar  to  sensor  imagery.  For  example,  some  aircraft 
and  rotorcraft  simulators  have  image-processing  capabilities  for  the  out-of-the-window 
(OTW)  view,  that  simulate  light  intensification,  FLIR  or  Radar  images  (e.g.,  De  Maio 
and  B^er,  1994). 

Various  hybrid  methods  have  also  been  used.  For  example,  in  FLIR  simulation,  some 
systems  have  combined  a  low-fidelity,  image  processed  background,  with  high-fidelity 
recordings  or  physical  models  of  targets  (e.g.,  De  Maio  and  Baker). 

Physics  based  simulation  tools  are  the  state-of-the-art  in  both  radar  and  FLIR  simulation. 
These  systems  use  digital  models  of  the  real  environment  (terrain,  atmosphere),  the 
sensor  system  (e.g.,  a  specific  FLIR)  and  implement  parameters  and  equations  of  the  real 
world  (e.g.,  the  physics  of  radiation)  (Blasband  and  Jafolla,  1999).  A  comprehensive, 
critical  review  of  physics-based  simulation  systems  is  presented  in  the  next  section. 

Physics-based  simulation  systems 

Sensor  simulation  tools  have  developed,  gradually,  along  with  technological  capabilities. 
Powerful  computers  and  3D  graphics  tools,  enable  the  formation  of  precise  and  detailed 
simulations.  Most  advanced  are  recent  physics-based  simulation  tools  that  model  the 
components  that  impact  real  sensor  images:  the  environment  (terrain  and  atmosphere), 
the  radiometric  equations,  and  the  sensor  system.  A  survey  was  conducted  to  identify 
existing  sensor  imagery  simulation  products  and  applications.  The  following  critical 
review  of  physics-based  simulation  tools  is  composed  primarily  of  a  group  of  simulation 
tools  built  by  or  in  collaboration  with:  MultiGen-Paradigm  (MP),  Surface  Optics 
Corporation  (SOC),  and  Photon  Research  associates  (PRA).  Other  simulation  tools,  built 
by  Technology  Service  Corporation  (TSC),  and  by  Camber  Corporation,  will  be 
mentioned  briefly.  The  written  references  to  these  simulation  tools  are  either  technical 
papers  or  the  companies’  web  sites: 

RadarWorks™  (MP)  -  A  physics-based  simulation  tool,  for  different  types  of 
imaging  radars.  It  is  comprised  of  two  main  software  components.  The  first 
component  (Radar  Vision),  generates  a  pixelized  RCS  map  of  the  desired  area,  in 
units  of  dBsm,  in  real-time.  The  second  component  (the  sensor  model),  models 
the  specific  radar  device  (Blasband,  Jorch,  &  Sigda,  1998). 
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Camber  Radar  Toolkit'^'^  (Camber  Corporation)  -  A  comprehensive  real-time 
radar  simulation.  Provides  a  complex,  real  world,  energy  level  model  of  the 
interaction  of  the  emitted  radio  transmissions  and  the  simulated  environment 
(http;//www.cambertx.com/RTK.html). 

RadBase™  (SOC)  -  A  physics-based  radar  database  generation  tool.  Calculates 
accurate  RCS  and  Amplitude  &  Phase  data  for  complex  targets  and  cultural 
features,  using  a  hybrid  geometrical/physical  optics  approach.  RadBase  has  been 
validated  against  range  measurements  (Blasband  &  Jafolla,  1999). 

XPATCH  (developed  for  the  US  government  by  DEMACO  [now  SAIC1)  -  An 
electromagnetic  computer  prediction  code  for  generating  RCS,  time  domain 
signatures  and  SAR  images  of  realistic  3-D  vehicles  (Andersh,  Lee  &  Ling  see 
http://www.demaco.com/papers/sbrl/). 

SensorVision'^''^  (MP)  -  A  real-time  physics-based  simulation  of  electro-optical 
sensor  imaging  (visible  through  far  infrared).  Produces  a  sensor  spectral  response 
(Anding,  1998). 

SensorWorks™  (PRA)  -  Adds  realistic  sensor  effects  to  the  SensorVision  image 
(McGlamery,  Johnson  &  Scully,  1998). 

IRGen®  (TSC)  -  creates  an  IR  database  as  viewed  by  a  thermal  IR  sensor,  using 
first-principles  models  of  heat  transport,  radiation,  and  sensor  processing 
(Technology  Service  Corporation,  1996). 

Physics-based  simulations  consist  of  many  components,  each  of  which  may  affect  the 
final  quality  of  the  simulation.  The  major  components  are  discussed  below. 

Modeling  of  the  environment  -  Database  construction 

A  digital  model  of  natural  terrain  and  of  human-made  objects  on  the  terrain,  is  the  first 
component  of  the  simulation.  The  model  serves  as  an  input  database  for  the  simulation 
tools.  Tools  for  building  digital  models  (e.g.,  MultiGen  Creator)  enable  the  user  to  go  to 
almost  any  desired  level  of  details.  Obviously,  the  level  of  details  may  affect  the  quality 
of  the  resulting  sensor  simulation,  hence,  the  decision  as  to  how  much  effort  should  be 
invested  in  the  digital  terrain  model,  may  impact  the  final  results. 

The  first  stage  in  building  a  digital  model  of  a  scene  is  modeling  the  structure  the  terrain, 
with  pixels  or  polygons.  Often,  a  model  of  a  real  terrain  is  most  desirable.  Using  aerial 
photographs  and  elevation  data,  precise  modeling  of  terrain  can  be  achieved.  One  major 
drawback  of  such  models  is  that  most  human-made  objects  do  not  have  elevation  data. 
Hence,  they  appear  two-dimensional,  and  without  special  treatment,  they  cannot  be 
treated  as  objects  in  the  simulation.  One  should  also  bear  in  mind,  that  the  resolution  of 
the  source  (e.g.,  aerial  photograph)  poses  an  upper  limit  to  the  resolution  of  the  sensor 
image.  For  example,  a  satellite  terrain  photograph  of  lOm/pixel  cannot  provide  higher 
than  lOm/pixel  details  of  the  terrain  (in  practice,  as  low  as  20m/pixel). 

Sensor  imagery  simulation  requires  not  only  the  structure  but  also  the  material  map  of  the 
terrain  and  the  terrain-objects.  As  will  be  shown  and  discussed  (in  further  sections),  this 
information  is  crucial  for  the  types  of  simulation  discussed  in  the  present  document. 

All  the  above  simulation  models  use  polygons  as  basic  building  blocks  for  terrain  and 
objects.  Polygons  can  be  assigned  different  properties,  like  color  and  texture,  which  can 
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make  them  look  similar  to  real-world  objects  to  the  naked  eye.  However,  in  order  to 
simulate  different  sensors,  additional  properties,  may  be  needed.  Hence,  the  next  stage  is 
to  assign  polygon  with  relevant  and  meaningful  properties.  When  dealing  with  SAR  and 
FLIR,  the  important  properties  relate  to  materials  and  their  characteristics,  e.g., 
reflectivity,  heat  emittance  properties,  etc. 

When  simulating  the  visual  band,  it  may  sometimes  be  possible  to  “save”  polygons  and 
use  texture  as  a  substitute  (e.g.,  a  building  may  be  represented  by  as  little  as  6  polygons, 
with  textures  of  bricks,  windows,  doors,  etc.).  However,  for  sensor  imagery,  polygons 
should  be  specified  in  accordance  with  material  boundaries.  It  may  not  be  satisfactory  in 
this  case  to  use  fewer  polygons,  while  “compensating”  with  textures. 

SensorVision™ 

SensorVision™  is  a  real-time  physics-based  simulation  program  tool  for  electro- 
optical  sensor  imaging  (visible  through  far  infrared).  It  produces  a  sensor  spectral 
response  of  both  out-the-window  (OTW)  displays  of  a  synthetic  environment  and 
correlated  sensor  views  of  these  environments. 

For  high-fidelity  results  of  the  simulation,  the  database  for  the  SensorVision™ 
simulation  tool  must  provide  an  accurate  representation  of  reality.  In  some  cases  it 
may  be  required  to  represent  some  specific  terrain,  while  in  others,  it  may  suffice  to 
provide  a  seemingly  “realistic”  terrain.  Polygon  representation  must  be  at  an  adequate 
level  of  detail  to  represent  slopes  of  surfaces  and  material  boundaries.  Reflectance 
and  thermo-physical  parameter  values  for  all  surfaces  must  be  accurate 
representations,  and  the  atmospheric  state  must  be  well  defined. 

The  synthetic  environment  can  include  digitized  natural  terrain  backgrounds,  cultural 
objects,  mobile  objects  and  atmospheric  states  that  are  homogeneous  in  the  horizontal 
dimension. 

In  conjunction  with  SensorVision™,  two  database  construction  tools  can  be  used  in 
the  pre-process  mode,  to  prepare  relevant  inputs  to  the  application: 

Texture  Material  Mapper™  (TMM) 

Generates  textures  of  material  codes  and  radiance,  by  transforming  RGB  colors  or 
intensities  to  linear  combinations  of  materials.  The  user  transforms  all  scene 
database  textures  into  the  material  codes  needed  for  the  simulation,  mapping 
visual  color  spots  to  materials,  and  indicating  the  substance  each  colored  spot 
represents.  TMM'^'^  includes  only  a  sample  of  materials,  but  it  can  be  expanded 
by  the  user.  The  user  has  the  responsibility  to  generate  a  relevant  and  valid 
database.  Reflectance  quantities  are  derived  from  the  materials  database,  which 
represents  the  most  commonly  occurring  measurable  case,  for  each  material.  It  is 
important  to  insert  correct  values  into  the  database:  either  valid  measured  data,  or 
representative  data,  depending  on  the  user’s  needs  and  on  the  availability  of  data. 

MOSART Atmospheric  TooF^  (MAT) 

Generates  material  surface  temperatures  and  atmospheric  quantities,  according  to 
parameters  chosen  by  the  user  (latitude,  longitude,  date  and  weather)  and  the 
simulated-sensor  spectral  response  function.  Thermal  quantity  (radiance 
parameters)  is  derived  from  the  material  surface  database.  MAT  does  not  provide 
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accurate  representations  of  high  vertical  vegetation  (e.g.,  trees)  or  heat  generators 
(e.g.,  operated  vehicles),  and  does  not  handle  water,  ice  and  snow. 

From  geometric  considerations,  SensorVision™  when  applied  to  high-fidelity 
databases,  can  provide  accurate  and  valid  radiance  images.  When  applied  to  databases 
with  fewer  details  than  the  pixel  resolution  and  spectral  response  of  the  simulated 
sensor,  simulation  fidelity  is  compromised.  SensorVision^”^  provides  the  user  with 
complete  control  over  the  desired  level  of  details,  and  over  database  construction.  In 
practice,  however,  the  user  may  not  have  access  to  sufficiently  detailed  information 
and  may  not  be  able  to  reach  the  desired  level  of  simulation  fidelity. 

IRGen® 

As  in  SensorVision™,  each  polygon  in  the  data  base  of  IRGen®  has  to  be  labeled 
with  a  material  code,  which  specifies  the  properties  of  that  surface.  The  material  code 
is  then  used  by  the  thermal  model  to  compute  surface  temperature  and  radiance.  In 
addition,  an  environment  model  contains  the  parameters,  which  specify  the  thermal 
environment  of  the  database. 

RadarWorks™ 

The  physics-based  approach  for  imaging  radar  simulations,  maps  scene  materials  to 
RCS  (Radar  Cross  Section)  values.  The  synthetic  environment  is  a  3-D  polygonal 
database  with  RGB  textures.  RadarWorks^M,  like  SensorVision™,  uses  the  TMM™ 
to  generate  material-coded  textures  from  visual  database  RGB  textures. 

RadBase™ 

RadBase™  is  designed  to  primarily  for  creating  detailed  and  accurate  models  of  3D 
objects.  The  input  to  RadBase™  is  a  3-D  wireframe  model  of  the  object  to  be 
simulated.  Like  the  other  simulation  inputs  described  above,  materials  have  to  be 
mapped  to  the  object  parts. 

Modeling  of  the  physics 

Once  a  terrain,  objects  and  environment  model  has  been  set,  the  physics-based  simulation 
system  subjects  it  to  radiometric  equations,  according  to  the  simulated  sensor  (e.g.,  heat 
emission  equations  for  FLIR;  electromagnetic  scattering  equations  for  SAR). 

SensorVision'*^'^ 

SensorVision™  evaluates  a  radiometric  equation  during  the  visualization  process,  for 
each  pixel  of  every  image  frame,  taking  into  account  the  input  databases.  The  output 
is  line-of-sight  radiance,  expressed  in  absolute  radiometric  units  (watts/cm2/sr),  for 
each  pixel  of  the  image. 

Users  can  configure  the  radiometric  equation  to  use  any  of  the  following  components: 

1 .  Diffuse  solar  and  lunar  reflections 

2.  Specular  reflections 

3.  Ambient  sky-shine  reflections 

4.  Thermal  emissions 

5.  Path  emission 

6.  Scattering 
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Effects  not  yet  included  are: 

1 .  Shadows  (reflective  or  thermal) 

2.  Luminosity  reflections  (sun/moon  and  sky) 

3.  Bi-directional  reflection  (diffuse  and  specular  component) 

The  process  can  be  run  in  real-time,  and  can  accommodate  dynamic  changes  to  the 
environment  or  the  view  perspective. 

Most  terrestrial  materials  are  very  diffuse.  For  them,  the  equation  is  a  good 
approximation. 

IRGen® 

Given  a  user-specified  thermal  environment,  IRGen®  computes  the  surface 
temperature  of  every  polygon  in  the  database.  This  is  a  first-principles  calculation 
using  the  time-dependent  heat  transport  equation,  integrated  with  a  finite-difference 
time-dependent  solution  method.  IRGen®  has  accurate  models  for  exterior  and 
interior  heat  sources  for  each  surface. 

The  thermal  model  computes  the  surface  radiance  from  the  surface  temperature, 
emissivity,  and  reflected  radiance  terms.  The  radiance  is  computed  in  the  sensor 
spectral  band,  and  later  serves  as  the  input  to  the  sensor  model. 

RadarWorks’^” 

The  RadarVision™  module  of  RadarWorks™  maps  scene  materials  to  RCS  values, 
taking  into  consideration  different  parameters  determined  by  the  user: 

1 .  Frequency  range  (l-27GHz) 

2.  Linear  polarization 

3 .  Ground  squint  angle 

4.  Radar  incident  angle 

5.  Pixel  resolution 

RCS  values  are  obtained  from  a  database,  and  through  interpolations. 

RadarWorks™  uses  two  databases.  The  first  is  a  database  of  natural  terrain  mean- 
backscatter-coefficients,  which  was  generated  from  validated  measures,  using  many 
different  radar  parameters.  The  second  is  a  database  of  RCS  values  for  cultural 
features,  as  a  function  of  frequency,  polarization  and  geometry.  The  primary 
databases  are  limited,  but  can  be  expanded  by  the  user. 

RCS  values  are  affected  by  material  information  (derived  from  the  database  built  with 
the  TMM'*'’^  -  [see  above]),  radar/polygon  geometry  (incident  angle  and  azimuth 
angle)  and  radar  parameters. 

Radar  shadows  are  computed  in  real-time,  according  to  the  radar/polygon  geometry. 

RadBase™ 

RadBase™  calculates  RCS  values  as  a  function  of  frequency,  polarization  and 
target/observer  geometry  (incident  angle  and  azimuth  angle),  using  the  physical  optics 
(PO)  approach  approximation  to  electromagnetic  scattering. 

PO  theory  is  based  upon  source  currents,  and  is  valid  in  cases  where  the  incident 
wavelength  is  much  smaller  than  the  length  of  the  object  scattering  the  energy.  It  uses 
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the  integral  equation  representation  for  the  scattered  fields.  It  relies  on  the  high 
frequency  assumption  that  the  scattered  field,  from  one  point  on  an  object  to  any  other 
point,  is  negligible  compared  to  the  incident  field.  Hence,  the  total  field  at  each  point 
on  the  surface  of  the  object  is  approximately  equal  to  the  incident  field  at  that  point. 
Thus,  the  equation  that  has  to  be  solved  is  much  simplified. 

RadBase™  includes  the  following  physical  phenomena:  blocking,  multiple-bounce 
interaction,  edge  diffraction,  polarization,  dielectric  materials  and  bi-static 
computations. 

The  calculations  of  RadBase™  are  based  on  the  model  that  serves  as  input,  and  on 
other  parameters  determined  by  the  user: 

1 .  Multiple  bounces:  the  number  of  multiple  bounces  off  the  object  that 
the  radar  beam  can  undergo  before  returning  to  the  receiver  -  currently, 
up  to  two  bounces  are  enabled. 

2.  Blocking:  refers  to  one  facet  of  the  target  vehicle  blocking  another 
facet  from  the  radar  view. 

3.  Edge  diffraction:  Edge  effects  can  have  significant  effect  on  the  RCS  of 
complex  objects.  RadBase™  enables  the  user  to  select  whether  or  not 
to  compute  edge  diffraction. 

4.  Maximum  Interior  Wedge  Angle  -  defines  the  maximum  angle  at  which 
two  facets  form  an  edge.  This  prevents  inclusion  of  edges  in  which  the 
interior  angle  is  close  to  180  degrees. 

5.  Frequencies:  the  frequencies  (GHz)  for  which  the  RCS  is  computed. 

6.  Elevation  angles:  the  elevation  angles  (degrees)  for  which  the  RCS  is 
computed. 

7.  Azimuth  angles:  the  azimuth  angles  (degrees)  for  which  the  RCS  is 
computed. 

The  output  is  a  file  that  contains  RCS  data  for  all  the  pre-defined  frequencies,  two 
polarizations,  and  pre-defined  elevation  and  azimuth  angles.  (A  different  file  contains 
the  amplitude  and  phase  data).  It  can  be  used  as  input  to  other  simulation  tools,  like 
RadarWorks,  to  obtain  a  precise  simulation  of  both  the  terrain  and  the  cultural 
features. 

XPATCH 

The  XPATCH  computer  code,  based  on  the  shooting  and  bouncing  ray  technique,  is 
used  to  calculate  the  polarimetric  radar  return  from  complex  geometric  shapes 
represented  by  CAD  (computer  aided  design)  files.  The  first-bounce  Physical  Optics, 
the  Physical  Theory  of  Diffraction,  and  the  multi-bounce  geometric  optic  ray 
contributions  are  included  in  the  equation. 

XPATCH  doesn’t  currently  perform  calculations  for  some  of  the  high  order  scattering 
effects  (traveling  and  creeping  waves,  surface  waves  and  resonant  effects). 

Modeling  of  the  sensor  system 

The  images  that  are  formed  by  implementing  valid  physical  equations  to  valid  digital 
models  are  “perfect”  sensor  images.  Perfect,  in  the  sense  that  their  quality  has  not  yet 
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been  diminished  by  the  sensor  itself.  Sensor  systems  introduce  noise  and  other  effects  to 
the  image,  and  are  bound  to  modify  its  appearance. 

SensorWorks’^'^ 

SensorWorks™  provides  the  capability  to  add  realistic  sensor  effects  to 
SensorVision™  images.  The  outcomes  reflect  the  effects  of  specific  electro-optical 
(EO)  sensors,  rather  than  an  OTW  view. 

In  a  basic  EO  sensor  system,  there  are  four  main  subsystems,  each  of  which  may 
affect  the  output  image  of  the  sensor: 

Optical  system:  Collects  light  from  the  scene,  and  forms  an  image  at  the  focal 
plane.  Design  and  manufacturing  errors,  which  can  cause  the  image  at  the  focal 
plane  to  be  blurred,  and  diffraction  blur  effects,  can  be  modeled. 

Focal  Plane  Array  (FPA):  An  array  of  one  or  more  detectors,  which  spatially 
sample  the  energy  in  the  scene  and  convert  it  into  a  signal.  A  single  detector 
channel  scans  the  scene  in  a  raster  fashion.  A  linear  array  of  detectors  scans  the 
image  in  one  direction,  and  a  two-dimensional  array  of  detectors  samples  the 
scene  without  scaiming.  Spectral  response,  detector  responsivity,  offset,  noise  and 
optical  blur,  which  are  produced  by  the  FPA,  can  be  modeled.  SensorWorks™ 
does  not  as  aecurately  model  detector  response  non-linearities  and  blooming. 

Signal  processor:  Converts  the  signal  to  a  useable  electronic  form.  For  purposes 
of  simulation,  the  signal  processor  can  usually  be  disregarded.  However,  the  DC 
restoration  and  non-uniformity  correction  processes,  if  significant,  should  (and 
can)  be  modeled. 

Display  system:  Generates  and  displays  an  image  of  the  converted  signal.  The 
display  has  a  modulation  transfer  function,  signal  transfer  function  and  non¬ 
linearity,  which  add  to  the  system  noise  that  can  be  modeled.  Operator  controls 
for  the  display  usually  add  random  LOS  errors  (drift  and  jitter),  that  can  be 
modeled.  Moreover,  the  operator  may  be  able  to  control  the  line-of-sight  of  the 
sensor  and  its  gain  and  offset  settings. 

Sensor  platforms  can  effect  the  sensor  image  by  motion  and  vibration.  Low  frequency 
motion  causes  jitter,  which  can  be  modeled.  High  frequency  motion  causes  blur, 
which  isn’t  modeled  by  SensorWorks™. 

A  large  number  of  individual  effects  can  be  combined  as  needed  to  simulate  the 
characteristics  of  a  wide  variety  of  sensors.  SensorWorks™  supports  the  following 
effects: 

Jitter  -  due  to  platform  motion. 

Blur  (Convolution)  -  due  to  optical-system  and  integration  of  the  scene  by  each 
sensor  detector-element  over  the  area  of  the  element. 

Sub-sampling  (Zoom-in)  -  depending  on  the  sensor. 

Saturation  Radiance  -  only  the  hard  saturation  is  modeled  (saturation  is  a  non¬ 
linearity  of  response  to  input  flux,  and  can  be  either  soft  or  hard). 

Responsivity  Variations  -  optical  sensors  have  pixel-to-pixel  variations  in 
responsivity.  The  arrays  can  be  customized  to  represent  different  sensors. 

Offset  Variations  -  for  either  intrinsic  offsets  or  the  residual  offsets  after  the  non- 
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uniformity  correction  (NUC)  process. 

Sensor  Noise  -  photon  and  eleetronic  noise.  A  single  noise  type  represents  the 
different  individual  noises.  Temporally  varying  random  noise  is  produced. 
Contrast  Control  (gain  and  offset)  -  manual,  auto  and  dynamic  range. 

White  hot  /  Black  hot  -  the  polarity  of  the  contrast  can  be  inverted. 

Zoom-out  -  to  increase  the  image  size  for  display  purposes. 

Display  color  -  for  simulating  display  systems  with  a  single  color  output. 

IRGen® 

The  sensor  model  contains  the  parameters,  which  specify  the  properties  of  the 
simulated  sensor.  Sets  of  sensor  parameters  can  be  stored  and  saved.  The  user- 
specified  sensor  parameters  include  spectral  band,  spectral  response,  display  dynamic 
range,  and  white  hot  /  black  hot  modes. 

RadarWorks’’’^ 

The  RCS  map,  which  is  the  output  of  the  RadarVision™  module,  serves  as  input  to  a 
radar  sensor  model.  The  output  is  presented  on  a  Plan  Position  Indicator  (PPI)  or  a 
raster  display  -  depending  on  the  radar  mode  being  simulated. 

The  user  selects  the  type  of  radar,  and  indicates  important  radar  parameters,  frequency 
and  polarization  effect  the  RCS  map. 

The  sensor  model  generates  a  radar  display  by  combining  the  RCS  map  with  the  user- 
defined  values  for  radar  parameters  (e.g.,  ground  squint  angle,  radar  incidenee  angle). 

RadarWorks  provides  several  radar  modes,  one  of  whieh  is  SAR.  The  SAR  mode 
uses  the  RCS  map,  which  represents  a  “perfect”  SAR  image,  as  input.  The  output  is  a 
modified  RCS  map  that  manifests  several  sensor  and  proeessing  effects  of  the  SAR. 

Motion  compensation:  Poor  motion  compensation  eauses  smearing  of  objects  in 
the  image.  The  amount  of  smearing  depends  on  the  quality  of  the  mathematical 
calculations.  The  SAR  model  provides  four  levels  of  motion  compensation. 
Frequency  agility:  Radar  returns  from  object  in  a  scene  are  affected  by  the  radar 
frequency  (scintillation).  Hence,  at  eertain  frequeneies,  a  single  pixel  may  appear 
brighter  than  nearby  pixels  of  the  same  material.  Recurring  looks  at  the  scene,  in 
different  frequencies  (known  as  frequency  agility),  averages  this  effect  and 
minimizes  the  speckle.  The  SAR  model  can  model  both  scintillation  and 
frequency  agility. 

SAR  mode:  Synthetic  aperture  radar  systems  usually  operate  in  one  of  two 
modes,  a  strip  mode  (suceessive  images  of  the  groimd),  or  a  spotlight  mode 
(higher  resolution  as  a  result  of  longer  integration  times).  RadarWorks™  is 
currently  restrieted  to  modeling  the  strip  mode. 

The  output  of  the  SAR  mode  depends  on  several  user  defined  parameters: 

1 .  Range  to  center  of  map 

2.  Squint  angle 

3.  Map  size 

4.  Resolution  method: 
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□  The  user  defines  resolution  in  meters/pixel,  or 

□  The  user  defines  FFT  and  PRF  and  RadarWorks  computes  the 
pixel  resolution 

5.  Motion  compensation 

6.  Frequency  agility 

7.  Nominal  aircraft  speed 
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Simulation  Fidelity  transfer  of  Experience  and  Object  Perception 

In  this  section  we  present  and  discuss  the  concepts  of  “simulation  fidelity”,  “transfer  of 
experience”,  “object  /  pattern  recognition”  and  some  related  topics.  These  concepts  are 
used,  in  the  next  sections,  to  analyze  simulated  sensor  imagery. 

Simulation  Fidelity 

The  concept  “simulation  fidelity”  is  used  to  describe  the  effective  differences  between  a 
real  and  a  simulated  system.  Ideally  the  simulated  image  should  be  identical  to  its  real 
world  counterpart  (i.e.,  perfect  simulation  fidelity).  However,  perfect  simulations  may  not 
be  feasible  (technically  or  financially)  and  may  not  always  be  necessary  (a  central  issue 
of  the  present  study),  therefore,  compromises  have  to  be  made. 

Ehret,  Gray  and  Kirshenbaum  (2000)  presented  the  concept  of  scaled  worlds  in  the 
context  of  field  research.  Based  on  Brehmer  and  Domer  (1993)  their  basic  supposition  is 
that  real  field  situations  are  too  complex  for  investigation  and  do  not  allow  definite 
conclusion,  whereas,  in  laboratory  studies  there  may  be  too  little  complexity  to  allow 
interesting  conclusions.  Hence  various  methods  of  simulation  have  to  be  developed  to 
bridge  between  the  worlds.  We  suggests  that  the  relevance  of  these  concepts  extends 
beyond  the  context  of  field  research  and  propose  to  use  them  in  the  analysis  of  other 
situations  in  which  a  real  world  is  represented  by  some  sort  of  simulation.  In  addition  to 
scaled  worlds,  in  which  scaling  is  based  on  a  cognitive  task  analysis,  it  is  possible  to  use 
high-fidelity  simulations,  synthetic  environments  and  microworlds.  Three  dimensions  are 
proposed  in  order  to  analyze  the  differences  between  these  types  of  simulations: 
tractability,  realism  and  engagement.  Tractability  examines  whether  the  user  or  the 
simulation  can  pursue  the  questions  of  interest  (i.e.,  research,  training,  etc.).  Tractability 
is  a  relative  dimension  defined  by  the  question  at  hand.  The  simulated  task  is  realistic  to 
the  extent  that  experiences  encountered  in  the  simulated  environment  occur  in  the  real 
world  task  environment.  Engagement  describes  something  about  users’  motivation,  e.g., 
is  the  simulation  interesting,  challenging,  etc.  This  dimension  may  be  relevant  to  the 
extent  that  experts  are  often  reluctant  to  use  low-fidelity  simulation  regardless  of  its 
purpose  and  relevance. 

Various  approaches  can  be  used  to  estimate  the  magnitude  of  differences  between  a  real 
and  a  simulated  image  (i.e.,  simulation  fidelity).  Three  such  approaches  are  discussed 
below. 


Visual  discrimination 

A  simple  and  straightforward  approach  is  to  use  a  measure  of  differences  between 
real  and  simulated  images  of  the  same  scene.  The  Samoff  Visual  Discrimination 
Model  (VDM  -  Lubin,  1995)  offers  an  advanced  technique  for  such  a  comparison. 
VDM  simulates  human  vision  mechanisms  for  discrimination  between  similar  images 
and  provides  a  quantitative  rating  for  the  differences.  The  model  is  designed  to 
compare  pairs  of  images.  The  input  to  the  model  is  a  pair  of  images  and  the  output  is 
a  map  showing  the  probability  that  an  observer  would  be  able  to  detect  the  differences 
between  the  images.  The  differences  may  be  presented  as  a  JND  (just  noticeable 
differences)  map,  in  which  the  high  gray  levels  correspond  to  high  probabilities  of 
discrimination  by  a  human  observer  and  vise  versa.  The  actual  probability  values  on 
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the  JND-maps  are  calibrated  in  terms  of  JNDs.  JND=1  corresponds  to  a  75% 
probability  that  a  human  observer  viewing  the  two  images  multiple  times  would  be 
able  to  see  the  differences;  JND=2  corresponds  to  93.75  probability;  JND=3  to  98.44, 
etc.  For  example,  Brickner,  Silbiger  and  Lubin,  (1998),  investigated  the  differences 
between  original  and  compressed  sensor  images.  The  threshold  of  discrimination 
between  compressed  and  uncompressed  images  was  determined  in  a  well-controlled 
psychophysical  experiment.  It  was  argued  that  image  compression  levels,  which  are 
below  discrimination  threshold,  could  be  used  without  consequences  on  operator 
performance.  If  such  a  comparison  would  have  been  made  between  a  real  and  a 
simulated  image  of  the  same  scene,  a  below-discrimination  threshold  difference 
between  the  images  could  be  interpreted  as  “perfect  simulation  fidelity”. 

This  approach  has  two  major  drawbacks.  First,  it  can  only  be  applied  to  images  that 
can  be  compared  pixel-by-pixel.  Secondly,  the  result  of  such  a  comparison  is  a 
number  that  indicates  the  magnitude,  but  not  the  significance,  of  the  differences. 
Clearly,  below  threshold  differences  between  real  and  simulated  images  indicate  very 
high-fidelity.  However,  perfect  fidelity  is  rarely  required  and  almost  never  feasible 
and  the  model  does  not  say  much  about  the  significance  of  larger  differences. 
Furthermore,  similar  JND  numbers  may  stem  from  different  types  of  differences 
between  images  and  have  different  effects  on  operators’  performance. 

Mathematical  models  of  detection,  recognition  and  identification 

Mathematical  models  of  target  detection,  recognition  and  identification  have  been 
developed  for  both  applied  (e.g.,  ATR;  Ross,  Bradley,  Hudson  and  O’Connor,  1999; 
Boshra  and  Bhanu,  1999)  and  theoretical  purposes  (e.g.,  Sheffer  and  Ingman,  1997). 
For  example  Sheffer  and  Ingman  (1997)  developed  a  model  ofr  analysing  ;issues 
involving  monop-spectral  target  recognition.  These  included  target  detection, 
recognition  and  identification  thresholds  and  predicting  the  functional  parametric 
dependencies  of  the  results  of  observation  experiments  by  human  observers.  An 
image  of  a  certain  scene  was  treated  as  a  sample  of  an  entire  set  of  images  of  that 
particular  scene.  A  difference  measure  called  the  Information  Difference  (InDif) 
between  two  image  sets  was  defined.  It  was  argued  that  aceomplishing  target 
recognition  tasks  is  equivalent  to  setting  thresholds  for  the  InDif.  The  applicability  of 
InDif  to  the  performance  of  the  human  visual  system  was  shown  both  analytically  and 
in  computer  calculations  of  noisy  images. 

In  can  be  argued  that  such  measures  could  be  used  to  express  the  similarity  between 
real  and  simulated  images.  If  an  image  of  a  simulated  scene  can  be  shown  to  represent 
a  set  of  real  images  of  a  similar  scene,  then  we  may  assume  a  certain  level  of  fidelity 
of  the  simulated  image.  Presently,  this  argument  is  purely  speculative  because  the 
model  has  been  applied  to  different  kinds  of  issues  and  has  not  been  validated  for  this 
type  of  problems. 

Transfer  of  experience 

A  more  viable  approach  to  the  evaluation  of  sensor-imagery  simulation-fidelity  is,  to 
identify  possible  differences  between  real  and  simulated  images  and  to  analyze  them 
in  terms  of  the  specific  objectives  of  the  simulation. 

This  approach  leads  to  the  concept  of  “transfer  of  experience”  (we  chose  to  use  this 
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generic  term  rather  than  the  more  common  “transfer  of  training”  because  simulation 
may  be  used  for  purposes  other  than  training).  The  question  is  to  what  extent 
experience  with  a  simulated  system  is  equivalent  to  similar  experience  with  a  real 
system.  In  simulator  training,  transfer  of  experience  was  quantified  (e.g.,  Roscoe, 

1971).  Perfect  transfer  means  that  one  hour  of  experience  with  the  simulated  system  is 
equivalent  to  one  hour  with  the  real  system;  fifty-percent  transfer  means  that  two  hours 
with  the  simulation  are  equivalent  to  one  hour  with  the  real  system,  etc.  Negative 
transfer  means  that  experience  with  the  simulation  produced  worse  results  on  real 
system  performance,  than  no  experience  at  all.  In  other  environments  (i.e.,  system 
research  and  development,  human  factors  studies  and  imagery  exploitation),  transfer  of 
experience  requires  specific  definitions  and  is  more  difficult  to  measure.  These  issues 
are  discussed  below. 

Pattern  and  object  recognition 

Less-than-perfect  simulation  fidelity  means  that  the  features  of  some  patterns  or  objects 
in  the  simulated  scene  are  represented  differently  than  in  the  real  scene.  In  order  to 
understand  these  differences  it  is  necessary  to  understand  the  mental  processes  of  object 
recognition.  Pattern  and  object  recognition  is  a  mental  process  during  which  a  perceived 
stimulus  is  matched  v^th  a  corresponding  internal,  mental  representation. 

Psychological  theories  of  object  recognition  may  be  classified  under  four  broad  headings: 
template  theories,  prototype  theories,  feature  theories,  and  structure-description  theories, 
all  of  which  are  classified  as  “bottom-up”  approaches.  In  addition,  “top-down”  theories, 
that  emphasize  the  effects  of  context  on  perception,  will  be  mentioned.  Distributed 
representation  and  connectionism  will  be  discussed  briefly.  Mathematical  model  for 
object  detection,  recognition  and  identification  have  been  discussed  briefly  in  section 
“Mathematical  models  of  detection,  recognition  and  identification”  above. 

Template  theories  argue  that  pattern  recognition  involves  matching  sensory  inputs  against 
specific,  labeled,  template-like  representations  stored  in  memory  (Neisser,  1967). 

Template  theories  were  rejected  at  an  early  stage,  because  they  failed  to  explain  some 
basic  aspects  of  human  perception,  e.g.,  the  ability  to  recognize  a  huge  variety  of  objects 
from  many  different  angles,  lighting  conditions,  etc. 

The  rigidity  of  template  theories  led  to  an  alternative  explanation:  prototype-matching 
theory.  Prototype  theory  argues  that  the  basic  elements  of  a  pattern  class  are  abstract 
memory  representations  called  prototypes.  A  prototype  is  characterized  as  a  synthesis  or  a 
statistical  average  of  all  the  individual  patterns  belonging  to  a  category  (Rosch,  1973). 

Feature  theories  postulate  that  the  visual  system  analyses  and  represents  sensory 
information  in  abstract,  primitive  information  units  called  features  or  attributes;  a 
“distinctive  feature”  can  be  used  to  make  a  critical  distinction  between  patterns  or  classes 
of  objects  (e.g.,  Gibson,  1969).  The  recognition  of  a  pattern  involves  the  analysis  of  its 
features  (e.g.,  line  direction,  size,  color,  etc.).  Feature  theories  received  strong 
reinforcement  from  neurological  studies  which  identified  brain  cells  sensitive  to  lines  of  a 
specific  orientation  (Hubei  and  Wiesel,  1963, 1968)  comers  and  angles  (DeValois  and 
DeValois,  1980).  In  addition,  highly  sophisticated  complex  cells  that  respond  to  specific 
complex  shapes  (e.g.,  hand  or  face)  were  discovered  (Shapley  and  Lennie,  1985). 

Biederman  (1987)  hypothesized  a  means  by  which  stable,  three-dimensional  mental 
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representation  of  objects,  can  be  based  on  simple  geometric  shapes  he  called  geons". 
According  to  Biederman’s  recognition-by-component  theory,  objects  are  recognized  by 
observing  their  edges  and  then  decomposing  them  into  geons,  which  can  then  be 
recomposed  into  alternative  arrangements. 

The  aforementioned  theoretical  approaches  may  be  characterized  as  bottom-up  theories 
that  interpret  perception  as  driven  by  the  data  that  constitutes  the  pattern.  These  theories 
do  not  fully  explain  context  effects.  Fairly  dramatic  context  effects  have  been 
demonstrated  by  Gestalt  theory  and  more  recently  by  many  other  researchers  (e.g., 
Pomerantz,  1981).  Ullman  (1989)  classified  the  different  paths  leading  to  object 
recognition  into  those  that  are  primarily  visual  and  those  that  are  supplemented  by  other 
sources.  Other  researchers  argued  that  objects  may  be  recognized  on  the  basis  of  their 
visual  features,  such  as  characteristic  shape  (Biederman,  1987);  color,  texture,  location, 
typical  motion  (Cutting  and  Kozlowsky,  1977);  and  also  on  the  basis  of  prior  knowledge, 
reasoning,  expectations  and  continuity  of  events  (Palmer,  1975). 

All  the  above  theories  represent  symbolic  approaches  to  mental  representation.  Their 
basic  view  is  that  human  cognition  is  centrally  dependent  on  the  manipulation  of 
symbolic  representations  by  various  rule-based  processes.  The  connectionist  approach 
proposes  an  alternative  view.  Connectionist  schemes  can  represent  information  without 
recourse  to  symbolic  entities  like  propositions;  they  are  said  to  represent  information  sub- 
symbolically  in  distributed  representations  (Smolensky,  1988).  Without  having  to  use 
large  sets  of  explicit  prepositional  rules  they  have  the  potential  to  model  complex 
behavior,  (e.g.,  Rumelhart,  McClelland  and  the  PDP  research  group,  1986).  A  distributed 
representation  does  not  have  symbols  that  represent  an  object  explicitly  but  rather  stores 
the  connections  strength  between  units  that  will  allow  sight  of  the  object  to  be  recreated. 
Input  cells  take  signals  from  vision  (vision  xmits),  a  neural  network  associates  the  pattern 
of  activation  that  arrives  at  the  various  vision  units.  The  distributed  representation  of  the 
sight  of  the  object  is  thus  represented  by  a  matrix  of  activation  over  the  imits  in  the 
network,  without  recourse  to  any  explicit  symbol  for  representing  that  particular  object. 
The  exposure  to  stimuli  leaves  “memory  traces”  that  stimulate  various  neural  xmits.  The 
connections  between  xmits  provide  the  “representation”  of  a  pattern  or  an  object.  These 
connections  are  strengthened  through  repeated  exposxire  and  weakened  during  periods  of 
no  exposure.  As  a  result,  objects  and  patterns  that  have  strong,  well-established 
connections  in  memory  are  recognized  rapidly  and  accurately,  whereas  weak  or 
inappropriate  connections  lead  to  slow  and  inaccurate  recognition.  In  other  words, 
transfer  of  experience  from  a  simulated  image  to  a  real  one,  depends  on  the  strength  of 
connections  that  developed  during  exposure  to  simulated  images.  Strong,  well-established 
connections  result  in  high  transfer,  poor  connections  result  in  low  transfer  and  irrelevant 
or  wrong  connections  may  result  in  negative  transfer  of  experience. 

Object  recognition  in  real-life  situations 

The  mental  representation  of  patterns  and  objects  is  a  theoretical  construct  that  cannot  be 
measured  directly  and  has  to  be  deduced  from  behavior.  We  suggest  that  whenever 
possible,  the  performance  of  pattern  /  object  detection,  recognition  and  identification  may 
serve  as  a  useful  behavioral  measure.  The  latest  version  of  the  vision  model  of  MIDAS  - 
the  computerized  human  performance  model  (Shively,  Burdick,  Brickner  and  Silbiger,  in 
preparation),  classifies  the  features  of  each  object  according  to  their  relative  contribution 


19 


DRAFT 

to  detection,  recognition  and  identification.  According  to  the  model,  detection  is  affected 
primarily  by  object-to-background  contrast,  movement,  motion  (movement  of  internal 
parts)  and  size.  Recognition  and  identification  are  affected  primarily  by  shape  (contour 
lines),  size,  movement  and  motion,  location,  color  and  texture. 

A  simulated  pattern  or  object  may  deviate  from  a  real  objet  in  any  of  these  features, 
thereby  affecting  its  perception.  The  effects  of  such  deviations  on  transfer-of-experience 
depend  on  the  nature  and  the  purpose  of  simulation.  For  examples,  contour  lines  are  a 
major  determinant  in  object  recognition  (Biederman,  1987;  Ullman,  1989),  hence,  low- 
fidelity  representation  of  contour  lines  may  affect  object  recognition.  False  representation 
of  contour  lines  may  lead  to  poor  or  even  negative  transfer  of  experience.  For  example,  if 
the  simulated  contour  lines  are  more  salient  than  the  real  ones,  the  operator  may  be  lead 
to  faultily  expect  easy  recognition  of  objects. 

Brickner  and  Zvuloni,  (1993)  investigated  the  preferred  polarity  in  FLIR  object 
recognition  tasks.  They  foimd  that  there  is  no  one  preferred  polarity,  rather,  some  objects 
are  more  readily  recognized  in  black-hot,  some  are  easier  to  recognize  in  white-hot,  while 
other  seem  to  be  insensitive.  The  researchers  determined  that  objects  are  easier  to 
recognize  when  their  presentation  is  similar  to  their  expected  regular  (visual  band) 
images  (e.g.,  it  is  easier  to  recognize  trees  as  dark  objects  on  a  light  background  rather 
than  the  other  way  round).  These  conclusions  may  also  be  generalized  to  simulated 
images.  Positive  transfer  of  experience  may  occur  when  the  expectations,  generated  by 
the  simulated  image  are  met  in  the  real  world  and  vice  versa. 

Maayan  (1989),  investigated  target  reeognition  in  low  quality  FLIR  images.  Based  on 
Biederman’s  (1987)  object  recognition  theory,  Maayan  argued  that  whereas  objects  in  the 
visual  band  are  recognized  primarily  by  their  contour  lines,  the  reeognition  of  objects  in 
the  IR  band  (FLIR)  relies  heavily  on  surface  determinants  (texture  and  internal  contrasts). 
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Sources  of  deviation  between  real  and  simulated  images 

In  this  section,  potential  deviations  between  real  and  simulated  images  are  analyzed.  The 
analysis  refers  to  state-of-the-art,  physics-based  simulation  models.  In  such  models, 
fidelity  deviations  may  stem  either  from  the  input  to  the  model  (i.e.,  databases  and 
parameters)  or  from  the  computations  of  the  model  itself.  As  shown  below,  state-of-the- 
art  models  are  potentially  capable  of  high-fidelity  computations.  Thus  the  major 
discrepancies  between  real  and  simulated  scenes  are  stem  from  the  input  data. 

The  first  sub-section  herein  deals  with  FLIR  systems  and  the  second  with  SAR  systems. 

Sources  of  deviation  between  real  and  simulated  FLIR  images 

Potential  sources  of  deviations  in  FLIR  images  are  presented  in  table  1 . 


Table  1:  Sources  of  deviations  between  real  and  simulated  FLIR  images 

Sources  of 
deviation 

Potential  deviations  between  real  and  simulated  FLIR  images 

General  simulation  fidelity 

All  physics  based  simulation  systems  are  based  on  a  simulated 
representation  of  the  real  world.  The  simulated  sensor  image  can  only  be 
as  good  as  this  representation.  For  example,  in  some  dynamic  simulation 
systems,  simulated  object-movement  may  seem  unnatural.  Such 
discrepancies  between  the  real  world  and  its  model  will  remain  in  the 
sensor-imagery  version  of  the  simulation  and  affect  its  quality. 

“Target”  objects 

Structure 

Level  of  detail  is  usually  expressed  in  terms  of  the  number  of  polygons  or 
other  facets.  Simple  models  may  contain  a  small  number  of  polygons. 
Additionally,  internal  parts,  including  heat-generating  parts  (e.g.,  engine), 
may  not  be  represented. 

Materials  & 
texture 

Materials  and  texture  can  be  defined  for  each  polygon.  However,  the  user 
may  not  have  access  to  a  sufficiently  accurate  and  detailed  material  and 
texture  model  of  the  object. 

Temperature 

Temperature  can  be  defined  for  each  polygon.  However,  the  user  may  not 
have  access  to  a  sufficiently  accurate,  time-based  model  of  temperatures 
of  the  object. 

Emissivity 

Current  simulation  systems  are  capable  of  accurate  computation  of 
emissivity  as  a  fimction  of  structure,  material,  texture,  and  temperature. 
Thus  its  resulting  accuracy  depends  primarily  on  the  former  variables. 

Conduction 

This  is  a  common  point  of  weakness  in  simulation  systems.  The 
conduction  of  heat  between  object  parts  is  often  not  represented  in  the 
models.  The  effect  may  be  of  particular  significance  for  internal,  heat 
generating  parts  (e.g.,  engines). 
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Reflectance 

IR  reflectance  from  most  solid  materials  is  small  and  should  not  have  a 
significant  impact  on  objects’  simulation  fidelity. 

Time  functions 

Changes  over  time  can  be  defined  in  advanced  models.  However,  realistic 
data  on  heat  emittance  changes  over  time  may  not  be  available. 

Background  (terrain) 

Terrain 

Real  terrain  can  be  sampled  using  topographical  data,  digital  terrain 
elevation  data  (DTED)  and  aerial  photography.  Simulation  fidelity 
depends  on  the  resolution  and  accuracy  of  these  sources  of  information. 

Terrain  objects 

Objects  that  were  included  in  the  aerial  photographs  of  terrain  (e.g.,  roads, 
buildings,  trees,  etc.),  do  not  have  elevation  data  and  are  presented  as  flat 
2D  patterns,  unless  treated  as  target-objects.  Full  3D  treatment  of  terrain 
objects  is  rarely  available. 

Materials  & 
textures 

Only  few  areas  are  mapped  in  detail  for  materials  and  textures. 

Typically,  the  fidelity  of  background  simulation  is  rather  low. 

Emissivity 

Emissivity  can  be  defined  for  each  polygon.  However,  accurate  emissivity 
mapping  of  terrain  are  rare,  limited  to  specific  areas  and  sometime 
proprietary  (limited  distribution).  Hence,  only  a  few  users  may  have 
access  to  limited  sets  of  emissivity  maps. 

Thermal  transfer 

Thermal  transfer  (conduction)  between  background  objects  is  usually  not 
represented  at  all  in  the  model.  However,  it  may  not  be  as  important  for 
the  terrain  as  it  is  for  heat  generating  target  objects. 

Reflectance 

Some  natural  objects  (e.g.,  water  surfaces)  reflect  IR  radiation.  Some 
simulation  devices  do  not  simulate  this  feature. 

Time  functions 

Changes  in  temperature  and  emissivity  can  be  fed  into  the  simulation. 
However,  the  intricate  changes  that  take  place  over  time  in  the  real 
environment,  cannot  be  represented. 

Atmosphere 

Particles 

Atmospheric  attenuation  of  radiation  can,  in  general,  be  presented. 
However,  the  specific  effects  of  various  types  of  particles,  in  different 
layers  of  the  atmosphere,  and  their  interaction  with  the  type  of  IR 
detector,  may  not  be  represented.  (Atmospheric  attenuation  models  of 
varying  levels  of  detail  exist,  e.g.,  the  number  of  individual  bands  of  the 
electromagnetic  spectrum  included  in  the  model.  A  trade-off  is  required 
however,  between  fidelity  of  the  atmospheric  model  and  computation 
time). 
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Platforms  and  FLIP  System 

Platforms 

The  platform  on  which  the  sensor  is  mounted  may  impact  the  line  of  sight 
between  the  platform  and  the  scene  (e.g.,  aircraft  vs.  tank)  and  the 
performance  of  the  sensor  system  (e.g.,  motion,  vibration,  etc.). 

Simulation  systems  may  not  be  designed  to  simulate  specific  platforms 
but  may  be  capable  of  simulating  some  relevant  effects.  These  effects 
may  have  to  be  fed  in  manually  (e.g.,  visual  angel,  jitter,  etc.). 

Detectors 

Different  FLIR  systems  use  different  detectors  for  the  same  range  or  for 
different  ranges  of  wavelengths.  Simulation  models  may  be  able  to 
specify  the  range  of  represented  wavelengths  (e.g.,  3-5  microns  V5.  8-12 
microns),  but  may  not  be  able  to  simulate  the  nuances  of  differences 
between  the  ranges  and  between  types  of  detectors  i.e.,  spectral 
sensitivity. 

Scanning  & 
signal  processing 

Various  scanning  methods  (serial,  vertical,  starring)  and  various  methods 
of  signal  processing  produce  different  visual  results.  Simulation  may  not 
reflect  the  results  of  specific  detectors  or  signal  processing  method.  Of 
particular  concern  is  the  representation  of  time  delay  integration,  which 
results  in  enhanced  signal-to-noise  imaging. 

Resolution 

The  final  resolution  of  the  image  is  a  product  of  both  spatial  resolution  of 
the  sensor  and  the  resolution  of  the  display  system.  Most  simulation 
system  may  support  the  definition  of  different  sensor  resolutions,  but 
simulation  of  the  display  is  often  ignored. 

Interactions 

Objects  / 
background 

A  major  drawback  of  most  physics-based  simulation  systems  is  that 
target-objects  and  the  terrain  background  are  defined  separately.  This  may 
have  several  consequences  and  constitutes  a  major  source  of  unrealistic 
FLIR  simulation  (e.g.,  lack  of  object  shadows  on  the  terrain,  salient 
contour  lines  of  objects,  lack  of  thermal  “smearing”  effects  due  to  high 
thermal  contrasts  between  targets  and  background). 

System  /  Objects 
&  background 

The  nature  of  interaction  between  targets  and  background  depends  also  on 
the  nature  of  the  sensor  system  (e.g.,  the  direction  of  scanning,  the  nature 
of  the  signal  restoration  processing).  These  effects  may  not  be  represented 
at  all  in  simulation  systems. 
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Sources  of  deviation  between  real  and  simulated  SAR  images 

Possible  sources  of  deviations  in  simulated  SAR  images  are  presented  in  table  2. 


Table  2:  Sources  of  deviations  between  real  and  simulated  SAR  images 


Sources  of 
deviation 

Potential  Deviations  Between  Real  and  Simulated  SAR  Images 

Generai  simulation  fidelity 

All  physics  based  simulation  systems  are  based  on  a  simulated  representation 
of  the  real  world.  The  simulated  sensor  image  can  only  be  as  good  as  this 
representation.  For  example  if  a  low-resolution  model  of  the  world  is  used, 
the  quality  of  SAR  simulation  may  also  be  effected.  This  factor  may  be  of 
particular  importance  in  the  simulation  of  high-resolution  SAR  systems 
(which  are  more  sensitive  to  details  than  low-resolution  systems). 

“Target”  objects 

Structure 

Level  of  detail  is  expressed  as  a  number  of  polygons  and  should  be  related  to 
the  resolution  of  the  simulated  SAR.  Simple  models  may  contain  a  small 
number  of  polygons,  which  may  suffice  for  simulating  low-resolution,  but 
not  for  high-resolution  systems.  In  particular,  the  accurate  geometry  of  parts 
with  high  RCS  (radar  cross  section)  should  (but  may  not  always)  be 
represented. 

Materials  & 
texture 

Materials  and  texture  can  be  defined  for  each  polygon.  However,  the  user 
may  not  have  access  to  a  sufficiently  accurate  and  detailed  material  and 
texture  maps  of  the  object. 

Radar  return 

Given  a  detailed  and  accurate  physical  model  of  the  object,  including: 
structure,  material  and  texture  and  a  valid  SAR  sensor  model,  the 
computation  of  radar  return  from  objects  in  the  scene  may  be  quite  accurate 
for  low  order  reflections.  However,  high  order  returns  (multi-bounce  ray 
contributions)  may  not  be  accurately  represented. 

Subsurface 

scattering 

Subsurface  reflections  and  their  associated  interactions  may  play  an 
important  role.  For  example,  the  RCS  of  a  treated  metal  surface  is 
determined  by  the  interactions  between  the  metal  and  the  treated  surface 
(subsurface  scattering).  Some  of  these  effects  may  be  represented  in  a 
generalized  manner  as  a  weighted-average  between  materials  (e.g.,  “paint  on 
metal,  “paint  on  wood”  in  RadarVision'^“). 

Transmitters 

Transmitters  (e.g.,  ground  radar)  may  produce  a  strong  “jammer-like”  signal, 
(depending  on  the  nature  of  the  systems).  In  some  simulation  systems,  this 
feature  is  not  be  represented. 

1  Background  (terrain)  | 

Terrain 

Real  terrain  can  be  sampled  using  topographical  data,  digital  terrain 
elevation  data  (DTED)  and  aerial  photography.  Simulation  fidelity  depends  ' 
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elevation  data  (DIED)  and  aerial  photography.  Simulation  fidelity  depends 
on  the  resolution  and  accuracy  of  these  sources  of  information. 

Terrain  objects 

Objects  that  were  included  in  the  aerial  photographs  of  terrain  (e.g,  roads, 
buildings,  trees,  etc.),  do  not  have  elevation  data  and,  imless  treated  as  target- 
objects,  are  presented  as  2D  patterns.  Full  3D  treatment  of  terrain  objects  is 
rare  and  hard  to  acquire. 

Materials  & 
textures 

Typically,  simulation  models  contain  or  provide  access  to  a  database  with 

RCS  representation  of  various  natural  materials  that  provide  a  general  “SAR 
look”.  Only  a  few  areas  in  the  world  are  mapped  in  detail  for  actual  materials 
and  texture.  In  addition,  the  representation  of  human-placed  objects  (e.g., 
roads,  houses  etc.),  may  be  deficient  or  lacking  (e.g.,  objects  represented  in 

2D  instead  of  3D). 

RADAR  System 

In  reality,  the  variety  of  radar  systems  is  enormous.  The  specific  parameters 
of  the  system  determine  the  nature  and  the  quality  of  the  resulting  image. 
Typical  SAR  parameters  that  are  represented  in  simulation  systems  include: 
range  to  center  of  map  (meters);  squint  angle  (degrees);  map  size  (meters); 
resolution  method  -  including  resolution  (in  meters/pixel)  and  FFT  and  PRF 
for  the  computation  of  pixel  resolution;  motion  compensation  (perfect  to 
high  error);  frequency  agility  (on/off);  aircraft  speed  (knots).  The  accurate 
representation  of  these  parameters  may  have  a  acute  effect  on  the  fidelity  of 
the  simulated  image  (depending  both  on  the  completeness  and  accuracy  of 
the  simulation  system  and  on  the  input  of  the  user). 
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Fidelity  requirement  and  validation  criteria 

In  theory,  some  of  the  state-of-the-art,  physics-based  simulation  programs,  are  capable  of 
producing  very  high-fidelity  simulations  of  sensor  imagery.  In  practice,  however, 
simulation  fidelity  depends  on  the  completeness  and  accuracy  of  all  components  of  the 
simulation,  i.e.,  the  representation  of  terrain,  terrain-objects  and  human-placed  objects, 
atmospheric  conditions,  the  radiometric  equations  and  the  representation  of  the  specific 
sensor  system.  As  we  have  shown  in  tables  1  and  2,  each  of  these  components  may  affect 
simulation  fidelity. 

The  efforts  required  to  produce  high-fidelity  simulations  may  be  tremendous.  In 
particular,  detailed  and  accurate  representations  of  terrain  material-maps  and  “terrain 
objects”  may  be  very  difficult  to  obtain  and  may  not  be  available  to  the  user  of  the 
simulation  system.  Therefore,  the  necessary  fidelity  of  the  simulation  should  be 
determined  by  the  requirements  of  the  application. 

The  technical  means  for  creating  high-fidelity  simulation  are  necessary  but  not  sufficient. 
One  must  also  know  what  to  create,  i.e.,  have  a  criterion  for  simulation  fidelity.  As  shown 
below,  good  criteria  may  be  hard  or  impossible  to  obtain. 

Fidelity  validation  criteria 

Types  of  validation  criteria 

We  have  defined  “fidelity”  in  terms  of  the  similarity  between  a  simulated  image  and  a 
reference  “real  image”.  There  may,  however  be  different  types  of  real  world  reference 
images.  The  following  distinctions  are  proposed: 

□  Specific:  the  reference  image  (real  image)  is  an  image  that  was  acquired  under 
specific  conditions  (e.g.,  specific  terrain,  atmosphere,  sensor  system,  etc.). 
Simulation  must  be  able  to  represent  many  different  and  highly  specific 
features. 

□  Generic:  the  reference  image  (real  image)  is  a  generic  or  prototypical  image 
representing  a  possible  or  a  typical  sensor  image  rather  than  a  specific  one. 
Simulation  is  required  to  represent  a  relatively  small  set  of  features. 

□  Vcuriable:  this  is  an  intermediate  category.  The  reference  image  (real  image)  is 
neither  highly  specific  nor  generic.  Simulation  is  required  to  represent  a  variety 
of  features  but  is  not  required  to  simulate  highly  specific  conditions. 

Availability  of  validation  criteria 

We  defined  the  “real  world  image”  as  the  validation  criterion  for  all  types  of 
simulated  image.  These  images  may  not  always  be  readily  available.  The  user  may 
have  access  to  various  recorded  or  real-time  sensor-imagery  that  may  be  used  as 
generic  or  as  variable  validation  criteria.  The  prospects  for  specific  validation, 
however  are  limited  and  depend  either  on  the  availability  of  good,  up-to-date 
intelligence,  or  on  the  existence  of  large  sets  of  real  world  images  from  which  specific 
combinations  can  be  drawn.  Furthermore,  in  many  cases  simulation  must  be  used 
because  the  real  image  does  not  yet  exist  (e.g.,  in  mission  rehearsal  and  sensor  system 
development)  and  therefore,  specific  validation  may  only  be  performed  upon 
completion  of  the  real  task  (performing  the  mission,  completing  system  development. 
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etc.). 

Required  Fidelity 

Required  fidelity  was  analyzed  in  terms  of  Bremer’s  (1992)  criteria  as  defined  above. 
Tractability  refers  to  the  ability  of  the  simulation  to  serve  its  defined  purpose,  represent 
the  right  data  at  the  right  grain  size,  with  the  right  time  stamp  and  also  to  its  usability. 
Tractability  is  a  relative  dimension  defined  by  the  purpose  of  the  simulation.  It  will  be 
defined  as  high  if  the  simulation  contains  most  of  the  elements  that  represent  the  real 
situation  and  vice  versa.  Realism  refers  to  the  extent  to  which  the  experiences 
encountered  in  the  simulated  environment  occur  in  the  real  task  environment.  In  the 
present  analysis  these  two  criteria  where  combined  into  the  term  “required  fidelity” 
where  tractability  denotes  the  aspects  that  have  to  be  simulated  for  any  given  purpose  and 
realism  refers  to  the  required  quality  of  these  aspects.  (Engagement,  the  third  criterion, 
deals  with  motivational  aspects  and  will  not  be  used  in  the  preset  analysis). 

Similarity  between  objects 

Fidelity  of  simulation  is  defined  by  the  similarity  between  a  simulated  and  real  object. 
Similarity  is  determined  in  terms  of  the  objects’  features:  size,  shape  (contour  lines), 
movement,  motion  (movement  of  internal  parts),  brightness,  color  (when  applicable) 
and  texture.  The  relations  between  objects  should  also  be  considered:  object  to 
background  contrast,  object  location,  order,  groups  of  objects,  etc. 

The  magnitude  of  each  of  these  dimensions  and  its  effects  on  perceived  similarity  are 
ill  defined.  Shively,  Burdick,  Brickner  and  Silbiger  (in  preparation)  have  argued  that 
detection  is  affected  primarily  by  object-to-background  contrast,  movement  and  size, 
while  recognition  and  identification  are  affected  primarily  by  shape,  size,  movement 
and  motion,  location,  color  and  texture.  Similarity  may  be  assessed  along  similar 
lines,  salient  features  that  determine  detection  may  contribute  to  large  differences 
between  real  and  simulated  objects,  whereas  smaller  features  that  determine 
recognition  and  identification  may  contribute  to  smaller  differences. 

Levels  of  fidelity 

The  level  of  required  fidelity  is  ranked  from  low  to  very  high.  It  expresses  the 
required  similarity  between  various  components  of  the  simulated  image  and  the 
reference  real-world  image.  Following  is  a  general  definition  of  each  of  the  fidelity 
levels. 

□  Very  high  -  the  simulated  entity  looks  very  similar  to  the  real  world  entity 
under  all  conditions. 

□  High  -  the  simulated  entity  looks  very  similar  to  real  world  entities  under 
different  conditions,  it  is  not  required,  however,  to  be  specifically  adapted  to 
all  possible  conditions. 

□  Medium  -  the  simulated  entity  look  similar  to  possible  real  world 
representations  of  that  entity. 

□  Low  -  the  main  features  of  the  simulated  entity  (e.g.,  size,  general  contrast) 
look  quite  similar  to  possible  real  world  representations  of  that  entity. 
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Types  of  applications 

In  this  section  we  propose  guidelines  and  analysis  tools  that  may  help  a  user  to  determine 
the  necessary  level  of  simulation  fidelity. 

The  analysis  refers  to  each  of  the  following  applications: 

□  Operator  training 

□  Mission  rehearsal 

□  System  research  and  development 

□  Human  factors  studies 

□  Imagery  exploitation 

Operator  training 

Training  of  sensor  imagery  users  can  take  place  in  different  environments  and  for 
different  purposes.  We  propose  to  distinguish  between  general  (procedural)  training  and 
specific  training. 

Validation  criteria:  the  simulated  image  should  always  be  validated  against  a  “real 
image”.  As  we  will  show  below,  sometimes  the  required  “real  image”  is  a  specific  image, 
acquired  under  specific  space,  time  and  sensor-system  conditions.  Whereas,  sometimes 
the  “real  image”  may  be  a  “possible”  sensor  image,  i.e.,  a  more  or  less  generic 
representation  of  a  sensor  image. 

Fidelity  criteria:  the  criterion  for  required  fidelity  in  ail  types  of  training  is  -  expected 
transfer  of  experience  (training)  from  the  simulation  to  the  performance  of  real  missions 
with  the  real  imaging  sensor  system. 


General  training 

By  “general  training”  or  “procedural  training”  we  refer  to  two  typical  situations.  First, 
the  simulation  may  be  used  for  introductory  or  general  acquaintance  with  a  family  of 
sensors  (e.g.,  8-12  micron  band  FLIR  systems).  Secondly,  there  may  be  a  training 
situation  in  which  sensor  imagery  is  incorporated  into  a  more  comprehensive 
simulated  environment  (e.g.,  a  flight  simulator,  where  sensor  imagery  is  part  of  a 
comprehensive  flight  training  setup). 
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Table  3:  Validation  criteria  and  fidelity  requirements  for  general  sensor  image  training. 


Type  &  availability  of 
Validation  criterion 

Rational  and  comments 

Generic  /  variable 

During  training  the  image  does  not  have  to  represent 
specific  conditions  but  rather  the  general  appearance  of 
the  specified  sensor  image.  However,  the  image  must 
represent  the  major  features  of  the  sensor  and  should 
represent  various  typical  conditions  and  difficulty  levels 
(e.g.,  images  at  good  and  bad  visibility). 

Easily  available 

Various  types  of  real  world  images  can  be  used  as 
criteria. 

Simulated 

components 

Required 

fidelity 

Rational  and  comments 

Target 

objects 

Medium 

Target  acquisition  difficulty  should  be  realistic 

Background 

Medium 

General  view  and  target  to  background  contrast  should  be 
realistic. 

Atmosphere 

Low 

(variable) 

Different  (realistic)  levels  of  visibility  should  be 
represented,  but  do  not  have  to  represent  specific  effects. 

System 

Medium 

The  image  should  represent  the  general  look  of  the 
relevant  sensor  system  (e.g.,  8-12  microns  FLIR  system). 

Interactions 

Low 

Fine  nuances  may  be  ignored. 

Specific  training 

“Specific  training”  refers  to  training  situations  that  are  aimed  primarily  at  acquiring 
and  improving  sensor  imagery  skills  (e.g.,  training  of  SAR  imagery  analysts,  FLIR 
weapon  system  operators,  etc.). 
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Table  4:  Validation  criteria  and  fidelity  requirements  for  specific  sensor  image  training 


Type  &  availability  of 
Validation  criterion 

Rational  and  comments 

Variable 

The  system  should  be  eapable  of  representing  a 
wide  variety  of  realistic  conditions.  It  may  not  be 
necessary  to  represent  every  possible  real  life 
situations  but  rather  a  representative  sample. 

Available 

A  variety  of  real  world  images  is  needed. 

Simulated 

components 

Required 

fidelity 

Rational  and  comments 

Target 

objects 

Medium  - 
very  high 

Depending  on  the  purpose  of  the  system.  For 
general  procedures  medium  fidelity  is  sufficient. 

For  target  acquisition  training,  very  high-fidelity  is 
desired. 

Background 

Medium  - 
high 

Depending  on  type  and  purpose  of  training.  For 
navigation  &  orientation  high-fidelity  is  required. 

For  target  aequisition  a  more  general  view 
representing  realistic  target-to-background  contrast 
may  suffice. 

Atmosphere 

Medium 

A  range  of  expected  atmospheric  conditions  and 
their  effect  on  image  quality  should  be  represented. 

It  may  not  be  necessary,  however,  to  simulate 
highly  specific  conditions. 

System 

High 

The  image  must  represent  the  look  of  the  relevant 
sensor  system  and  the  specific  features  of  different 
systems  (whenever  relevant). 

Interactions 

Medium  - 
high 

Fine  nuances  in  the  image  may  be  important, 
especially  for  expert  operators. 

Mission  rehearsal 

Mission  planning  and  mission  rehearsal:  If  a  mission  environment  can  be  simulated,  the 
operator  may  use  it  to  plan  the  mission  and  rehearse  it  before  actual  performance. 

Validation  criteria:  the  simulated  image  should  be  validated  against  a  real  image  expeeted 
during  the  mission. 

Fidelity  criteria:  the  eriterion  for  required  fidelity  is  expected  transfer  of  experience  from 
rehearsal  with  the  simulation  to  the  performance  of  the  real  mission. 
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Table  5:  Validation  criteria  and  fidelity  requirements  for  mission  rehearsal 


Type  &  availability  of 
Validation  criterion 

Rational  and  comments 

Specific  -  variable 

The  system  should  be  able  to  simulate  the  expected 
conditions  during  the  mission.  Otherwise,  negative 
transfer  may  result.  If  such  realism  is  not  feasible,  the 
system  should  be  used  for  training  (i.e.,  improving 
general  skills)  rather  than  for  specific  mission  rehearsal. 

Hardly  available 

The  real  world  images  exist  only  after  mission 
completion.  Prior  validation  must  be  based  on  the 
prediction  of  expected  conditions. 

Simulated 

components 

Required 

fidelity 

Rational 

Target 

objects 

High 

Targets  should  look  as  similar  as  possible  to  their 
expected  representation  in  the  real  world. 

Background 

High- 

medium 

Background  should  look  similar  to  its  expected 
representation  in  the  real  world.  The  actual  importance  of 
this  factor  depends  on  the  type  of  simulated  sensor  and  on 
the  type  and  variability  of  actual  terrain.  (I.e.,  do  terrain 
features  play  an  important  role  in  mission  performance?). 

Atmosphere 

High- 

medium 

Expected  levels  of  visibility  and  potential  sources  of 
noise  and  interference  should  be  represented.  If  unknown 
various  conditions  can  be  used  as  a  representative  sample. 

System 

High 

The  image  must  represent  the  specific  look  of  the  relevant 
sensor  system. 

Interactions 

High- 

medium 

The  interaction  between  major  simulation  components 
and  fine  nuances  may  determine  the  specific  image  and 
play  a  significant  role  during  the  real  mission. 

Note:  In  some  mission  rehearsal  applications  both  high  and  medium  fidelity  simulation 
may  be  appropriate.  While  high-fidelity  simulation  may  help  in  preparing  for  the  details 
of  the  mission,  medium  fidelity  simulation  may  support  general  orientation  in  the  area  of 
interest  and  establish  a  spatial  frame  of  reference.  It  is  crucial,  however,  for  the  user  to  be 
aware  of  simulation  fidelity  limitations,  otherwise,  negative  transfer  of  experience  may 
occur. 

System  research  and  development 

During  system  research  and  development,  various  system  components  or  whole  systems, 
may  be  simulated  in  order  to  test  concepts  and  ideas  at  an  early  stage.  For  example: 
Sensor  system  design,  sensor  system  analysis,  sensor  performance  analysis  and 
evaluation  and  automatic  target  recognition  (ATR)  development,  research  and  training.  In 
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many  cases  simulation  serves  engineering  purposes  and  may  not  depend  on  the  specific 
nature  of  the  final  image.  We  will  focus  on  applications  that  bear  on  human  performance 
and  specifically  on  ATR.  (ATR  algorithm  development  may  employ  large  amounts  of 
simulated  imagery,  including  many  targets,  imaging  geometry’s  and  backgrounds,  in 
order  to  “train”  the  algorithm  to  robust  performance  over  a  wide  range  of  scenario 
conditions). 

Validation  criteria:  the  simulated  image  should  be  validated  against  the  real  image 
expeeted  in  the  real  system  under  the  same  specified  conditions  and  parameters. 

Fidelity  criteria:  the  criterion  for  required  fidelity  is  expected  transfer  from  simulation 
results  to  the  design  of  the  real  system. 

Table  6:  Validation  criteria  and  fidelity  requirements  for  system  research  and 
development 


Type  &  availability  of 
Validation  criterion 

Rational 

Specific 

The  simulated  system  must  be  able  to  simulate  the 
expected  products  of  real  system  under  the  same  working 
conditions.  Low-fidelity  simulation  may  result  in  erroneous 
design  guidelines  for  the  real  system. 

Available  -  hardly 
available 

Criteria  may  be  available  for  modifications  of  existing 
systems  or  existing  components  and  may  not  be  available 
for  new  systems  and  components. 

Simulated 

components 

Required 

fidelity 

Rational 

Target 

objects 

High  -  very 
high 

Depending  on  the  purpose  of  simulation.  E.g.,  for  ATR 
simulation,  very  high  simulation  fidelity  of  target  objects  is 
required.  This  may  include  articulation  of  parts  of  the 
object  (turret,  guns,  hatches,  etc.)  and  the  presence  of 
variations  (e.g.,  externally  mounted  fuel  cans). 

Background 

High 

Depending  on  the  purpose  of  simulation.  E.g.,  background 
components  may  play  a  very  important  role  in  geological 
surveys. 

Atmosphere 

High 

Depending  on  the  purpose  of  simulation.  E.g.,  for  ATR 
specific  atmospheric  conditions  may  affect  performance 
and  should  be  represented. 

System 

Very  high 

Depending  on  the  purpose  of  simulation,  simulation  may 
often  be  system  specific. 

Interactions 

High 

Depending  on  the  purpose  of  simulation.  E.g.,  for  ATR, 
target  /  background  parameters  and  atmospheric  parameters 
may  interact  and  create  specific  and  unexpected  results. 
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Human  factors  and  Imagery  exploitation  studies 

Performance  of  controlled  human-performance  studies  in  the  field  is  challenging  and 
sometimes  impossible.  Simulated  laboratory  conditions  provide  very  useful  substitutes,  at 
least  for  preliminary  studies.  Some  examples  for  the  use  of  simulated  sensor  imagery  are: 

□  Studying  various  image  enhancement  techniques  and  their  effects  on  imagery  analyst 
performance  (e.g.,  Kuperman,  Brickner  and  Nadler,  1998) 

□  Generate  and  evaluate  softcopy  imagery-interpretation-key-materials  for  use  by 
imagery  analysts. 

Validation  criteria:  simulated  imagery  must  provide  a  representative  sample  of  the  real 
situation  that  is  being  simulated.  The  actual  criterion  depends  on  the  specific  purpose  of 
the  study. 

Fidelity  criteria:  the  criterion  for  required  fidelity  is  the  expected  transfer  fi-om  simulation 
results  to  real  behavior  results  (i.e.,  valid  generalization  of  research  results). 

Table  7:  Validation  criteria  and  fidelity  requirements  for  human  factors  and  imagery 
exploitation  studies 


Type  &  availability  of 
Validation  criterion 

Rational 

Variable  -  Specific 

The  simulated  image  must  provide  a  representative  sample 
of  investigated  effects,  and  yield  similar  human  behavior. 
Since  this  may  be  hard  to  determine,  it  is  recommended  to 
rely  on  actual  similarity  between  a  simulated  and  a  real 
image.  Level  of  specificity  depends  on  study  purpose. 

Available  -  hardly  available 

Criteria  may  be  available  for  research  on  existing  systems 
and  may  not  be  available  for  research  of  new  systems  and 
components. 

Simulated 

components 

Required 

fidelity 

Rational 

Target 

objects 

Medium  - 
very  high 

Depending  on  the  purpose  of  simulation,  e.g.,  for  target 
acquisition  studies  it  should  be  very  high. 

Background 

Medium  - 
High 

Depending  on  the  purpose  of  simulation,  e.g.,  for  studying 
background  clutter  effects,  it  should  be  high. 

Atmosphere 

Low  -  high 

Depending  on  the  purpose  of  simulation,  e.g.,  for  studying 
visibility  effects,  it  should  be  high. 

System 

Low  -  Very 
high 

Depending  on  the  purpose  of  simulation,  i.e.,  study  of  a 
specific  system  vs.  study  of  a  general  phenomenon. 

Interactions 

Medium  - 
High 

Depending  on  the  purpose  of  simulation,  interactions  may 
affect  some  parameters  that  are  important  for  the  study. 
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Initial  Validation  of  Concepts 

The  concepts  and  the  proposed  simulation  fidelity  evaluation  criteria,  were  examined 
with  the  help  of  some  subject  matter  experts  (SME). 

Method 

Subject  Matter  Experts 

Eight  male  SME  (expert  sensor  imagery  users)  underwent  a  structured  interview 
based  on  the  checklist  described  below, 

•  Four,  Israel  Air  Force  (lAF)  reserve  SAR  photo  interpreted,  were  paid  for 
their  participation.  These  subjects  had  an  average  of  3  years  of  experience  with 
SAR  imagery.  One  of  them  also  had  1.5  years  of  experience  with  FLIR 
systems. 

•  Two  lAF  pilots  with  both  operational  and  research  FLIR  and  SAR  experience, 
participated  voluntarily.  They  had  an  average  of  6  years  of  experience  with 
FLIR  and  3  years  of  experience  with  SAR  systems. 

•  Two  lAF  reserve  RPV  operators  were  paid  for  their  participation.  They  had  an 
average  of  2.5  years  of  FLIR  experience. 

The  checklist 

A  checklist  was  designed  as  a  qualitative  validation  tool  that  allows  the  SME  to 
evaluate  the  fidelity  of  given  simulation  products.  The  checklist  is  presented  in 
appendix  I. 

The  checklist  contains  the  following  sections: 

•  An  introduction  describing  the  objectives  of  the  present  research  and  defining 
the  concepts  that  are  being  used  in  the  checklist. 

•  Personal  details  of  the  SME  including  expertise  with  relevant  sensors  and  a 
short  description  of  related  missions. 

•  Checklists  for  the  evaluation  of  the  required  simulation  fidelity  of  various 
components  in  a  simulated  FLIR  or  SAR  system. 

Procedure 

Based  on  the  checklist,  a  structured  interview  was  conducted  individually  with  each 
SME.  Each  interview  lasted  2.5  -  3  hours. 

The  SME  was  briefed  on  the  objectives  of  the  research  program  and  introduced  to 
concepts  of  physics-based  simulation,  components  of  a  simulated  image,  simulation 
fidelity,  validation  criteria  and  levels  of  required  simulation  fidelity. 

Then  his  personal  details  were  recorded,  including  military  occupation,  familiarity 
and  experience  with  relevant  sensor  systems,  and  the  nature  of  missions  performed 
with  each  sensor. 

The  SME  was  then  asked  to  select  a  task  that  can  be  performed  with  the  simulated 
version  of  a  familiar  sensor  system.  Each  subject  referred  to  two  different  sensor 
systems  (e.g.,  FLIR  and  SAR);  or  to  two  different  missions  with  the  same  system 
(e.g.,  reconnaissance  and  photo  interpretation)  or  to  two  different  objectives  for  the 
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usage  of  the  same  simulation  (e.g.,  training  and  mission  rehearsal).  The  researcher 
filled  in  the  checklist  and  recorde  the  SME’s  comments. 

The  types  of  systems,  tasks  and  purposes  of  simulated  application  that  were 
investigated  over  all  SME,  are  presented  in  table  8. 

Table  8:  The  types  of  systems,  tasks  and  purpose  of  simulated  application  that  were 
investigated.  Numbers  in  cells  indicate  the  number  of  cases  in  the  specified  category. 


Mission 

Photo  interpretation 

Reconnaissance 

Target  recognition  & 
designation 

Simulation 

application 

Training 

Mission 

rehearsal 

Training 

Mission 

rehearsal 

Training 

Mission 

rehearsal 

SAR 

4 

1 

1 

- 

1 

1 

FLIR 

1 

1 

2 

- 

3 

1 

Results  and  discussion 

In  general,  the  method  proved  to  be  quite  effective.  The  SME  had  no  difficulty  in 
understanding  the  concepts  and  relating  to  a  hypothetical  simulated  task  in  their  areas  of 
expertise. 

The  average  results  for  collapsed  groups  of  cell  (from  table  8)  are  presented  in  appendix 
II.  Some  of  the  major  comments  and  insights  are  presented  below.  It  should  be  noted 
however,  that  these  averages  do  not  have  real  statistical  meaning,  not  only  because  of  the 
small  samples  but  also  because  each  SME  had  his  idiosyncratic  definition  of  the  nuances 
of  the  simulated  task.  Nevertheless,  the  ratings  of  required  fidelity  were  quite  consistent 
and  tend  to  support  our  theoretical  analysis.  The  SME  comments  provide  some  insights 
regarding  the  relative  importance  of  various  components  of  a  simulated  sensor  image. 

Some  of  the  major  points  are  summarized  below. 

SAR  imagery 

Fidelity  recommendations 

1 .  SAR  in  general  (and  in  particular  the  SAR  systems  that  were  used  by  the 
SME)  are  relatively  low-resolution  systems.  As  a  result,  most  objects  catmot 
be  recognized  solely  by  their  radar  signature  and  require  supporting 
information.  This  information  may  come  from  various  sources,  for  example: 
patterns  of  objects  (e.g.,  SA2  missile  site);  spatial  relations  with  large,  easy-to- 
recognize  objects  (e.g.,  objects  in  an  airfield);  known  geographical  location 
(e.g.,  is  an  object  present  or  absent  in  a  previously  known  location).  The 
required  fidelity  of  various  objects  in  a  simulated  image  reflects  these 
relations.  Radar  scattering  should  maintain  correct  size  and  magnitude 
proportions  between  various  types  of  objects,  but  (depending  on  system 
resolution)  it  may  not  be  necessary  to  provide  highly  accurate  structure 
details. 

2.  Fidelity  requirements  of  terrain  and  background  objects  depend  on  their 
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expected  role.  While  many  objects  are  of  low  importance  and  may  be 
represented  as  texture  (e.g.,  forest,  rocky  ground),  some  key  objects  are  highly 
important  and  require  high-fidelity  simulation.  The  relative  importance  of 
objects  is  task  specific,  often,  however,  human-made  structures  (e.g.,  roads, 
airfields,  power  lines,  defense  positions,  trenches,  etc.)  may  be  of  high 
importance.  Changes  to  the  natural  terrain  that  are  not  represented  in  the  DTM 
(e.g.,  defense  positions,  trenches)  are  most  difficult  to  simulate. 

3.  Atmospheric  conditions  have  little  or  no  effect  on  SAR  and  do  not  have  to  be 
simulated. 

4.  Any  SAR  image  is  highly  dependent  on  the  features  of  the  specific  system  that 
is  being  used.  Depending  on  the  purpose  of  the  simulation  it  may  be  important 
to  provide  high-fidelity  simulation  of  sensor  specific  effects.  I  addition,  it  may 
be  important  (particularly  for  training  purposes)  to  teach  operators  how  to 
identify  image  quality  defects  (e.g.  blur).  For  these  purposes,  specific  sensor 
and  platform  effects  should  be  simulated. 

5.  Interactions  between  features  in  the  SAR  imagery  are  marginal  and  do  not 
have  to  be  simulated. 

Applications 

The  SME  recommended  using  simulated  SAR  imagery  for  basic  and  advanced 
training  of  photo  interpreters  and  airborne  reconnaissance  operators. 

For  mission  rehearsal,  however,  they  thought  that  it  might  be  virtually  impossible  to 
create  sufficiently  high-fidelity  simulation.  Two  SME  recommended  using  regular 
images  (in  the  visual  band)  for  learning  the  terrain,  combined  with  independent  SAR 
images  of  relevant  objects  for  familiarization  with  objects-of-interest,  patterns  of 
objects,  key  ground  features,  etc. 

FLIR  imagery 

Fidelity  recommendations 

1 .  Target  recognition  is  the  major  task  in  all  missions  that  were  investigated  in 
the  present  validation  study.  Therefore,  it  is  most  important  to  provide  high- 
fidelity  simulation  of  target  objects,  including  structure,  materials  and  texture, 
temperature  and  emissivity.  It  is  also  important  to  represent  the  variety  of 
possible  or  at  least  probable  thermal  signatures  of  targets,  including  changes 
over  time  and  activity  related  changes  (e.g.,  engine  operating  or  not).  Heat 
conduction  between  parts  of  the  same  objects,  (e.g.,  fi'om  engine  to  body)  is 
also  of  some  importance  because  it  creates  the  continues,  natural-looking 
representation  of  objects.  This  feature,  however,  is  beyond  the  capability  of 
most  simulation  systems. 

2.  For  training  applications,  it  is  quite  important  to  have  a  reasonably  accurate 
and  realistic  looking  representation  of  the  terrain,  however,  it  may  not  be 
necessary  to  represent  every  little  detail  or  to  represent  highly  realistic  thermal 
features  of  each  object. 
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3.  High-fidelity  simulation  may  only  be  important  for  objeets  that  play  a 
significant  role  in  the  specific  task  at  hand.  Similar  to  SAR,  some  human- 
made  objects  and  features  may  be  important  for  orientation  and  general 
understanding  of  the  image.  Some  of  these  objects  may  be  relatively  easy  to 
simulate  (e.g.,  flat  objects  like  roads,  or  symmetrical  structures  like  buildings 
and  power  poles),  while  others  may  be  very  hard  to  simulate  unless  they  are 
represented  in  the  DTM  (e.g.,  trenches,  defense  positions).  The  effect  of 
objects  on  their  background  (e.g.,  armored  vehicle  tracks)  is  an  important 
feature  that  is  beyond  the  capability  of  most  simulation  systems. 

4.  Atmospheric  conditions  have  significant  effects  on  FLIR.  For  training 
purposes  it  may  be  desired  to  represent  different  levels  of  visibility.  It  may  not 
be  necessary,  however,  to  represent  highly  specific  effects  of  various  particles. 

5.  Various  types  of  FLIR  may  differ  considerably  from  each  other,  therefore,  it  is 
important  to  provide  a  high-fidelity  representation  of  the  sensor  of  interest. 

6.  Interactions  between  the  target  and  the  background  may  be  highly  important 
for  the  understanding  of  the  image,  however,  as  indicated  in  section  3  above, 
high-fidelity  representation  of  such  interaction  may  be  impossible. 

Applications 

All  participating  FLIR  experts  thought  that  high-fidelity  simulated  FLIR  can  be  very 
usefiil  for  training  of  various  types  of  operators  at  various  levels  of  expertise  (e.g., 
RPV  operation,  weapon  system  operation,  tactical  airborne  reconnaissance, 
surveillance  and  target  acquisition  applications). 

Effective  simulation  must  be  capable  of  representing  both  static  (e.g.,  specific  thermal 
signatures)  and  dynamic  processes  (e.g.,  changes  over  time).  The  possible  usage  of 
simulated  FLIR  for  mission  rehearsal  was  controversial  among  the  SME.  Some 
thought  that  the  prospects  of  viewing  an  unknown  area  prior  to  a  mission  are  very 
attractive.  However,  given  the  level  of  complexity  and  variability  of  FLIR  images, 
most  SME  doubted  whether  it  might  be  possible  to  create  sufficiently  high-fidelity 
simulation  for  specific  purposes  (i.e.,  create  a  simulated  image  similar  to  the  real 
image  expected  during  the  mission).  Concern  was  expressed  that  low-fidelity 
simulation  may  lead  to  negative  transfer  of  performance.  Hence,  if  high-fidelity 
simulation  eannot  be  achieved,  it  may  be  better  to  avoid  full  simulation  and  use  part 
simulation  for  viewing  thermal  signatures  of  expected  objects,  anticipate  weather  and 
time  effects  on  the  sensor,  etc. 
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Appendix  I  -  The  Checklist 

Introduction  and  Objectives 

The  checklist  is  intended  for  the  conduction  of  structured  interviews  with  subject  matter 
experts  (SME),  who  regularly  use  SAR  or  FLIR  images  a  s  part  of  their  job.  It  is  designed 
to  evaluate  the  required  fidelity  of  simulated  SAR  and  FLIR  sensor  images,  for  various 
types  of  users  and  various  purposes.  Fidelity  is  defined  in  terms  of  the  similarity  between 
a  simulated  image  and  a  reference  “real  image”. 

First,  let  us  introduce  you  to  concepts  related  to  the  analysis  of  sensor-system  simulation. 

□  Real  and  simulated  sensor  images. 

□  Physics  based  simulation 

□  Components  of  physics  based  simulation:  terrain,  objects  /  targets,  atmosphere, 
sensor  system,  interactions  between  components. 

□  Simulation  fidelity  -  sources  of  deviation  between  real  and  simulated  images;  general 
simulation  fidelity,  targets  and  other  objects,  background  (terrain),  atmosphere, 
platform  and  sensor  system,  interactions  between  components . 

Definition  of  concepts 

Types  of  validation  criteria 

•  Specific:  the  reference  image  (real  image)  was  acquired  under  specific 
conditions  (e.g.,  specific  terrain,  atmosphere,  sensor  system,  etc.). 

•  Generic;  the  reference  image  (real  image)  is  a  generic  or  prototypical  image 
representing  a  possible  or  a  typical  sensor  image  rather  than  a  specific  one. 

•  Variable:  an  intermediate  category.  Simulation  is  required  to  represent  a 
variety  of  features  but  not  to  simulate  highly  specific  conditions. 

Levels  of  fidelity 

•  Very  high  -  the  simulated  entity  looks  very  similar  to  the  real  world  entity 
under  all  conditions. 

•  High  -  the  simulated  entity  looks  very  similar  to  real  world  entities  under 
different  conditions,  but  is  not  required,  simulate  all  possible  conditions. 

•  Medium  -  the  simulated  entity  look  similar  to  possible  real  world 
representations  of  that  entity. 

•  Low  -  the  main  features  of  the  simulated  entity  (e.g.,  size,  general  contrast) 
look  quite  similar  to  possible  real  world  representations  of  that  entity. 
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Personal  details  and  level  of  professional  experience 

Name: _ 

Age: _ 

Gender: _ 

Military  occupation: _ 

Types  of  sensor  systems:  _ 


Time  and  level  of  experiences  with  each  system; 


System 

Time 

Level 

The  simulated  sensor  system  (to  be  evaluated) 

Name  of  system _ 

Type  of  system _ 


The  simulated  mission 

Name  of  mission: _ 

Description  of  the  mission: 
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Required  simulation  fidelity  (analytical) 


Type  of  required  validation  criteria:  Specific  /  Variable  /  Generic  / 
Availability  of  Validation  criterion:  High  /  Low/ None 


Table  9:  Rational  and  comments  -  FLIR  systems 


Simulated 

components 

Sub 

Components 

Required 

fidelity 

Rational  and  comments 

Target 

objects 

Structure, 
Materials  & 
texture, 
temperature, 
&  emissivity 

Conduction 

Reflectance 

Time  & 

changing 

conditions 

Background 

Terrain 

Terrain 
objects 
Materials  & 
textures, 
emissivity 

Thermal 

transfer 

Heat 

generating 

objects 

Reflectance 

Time 

functions. 

Atmosphere 

Particles 

Platform 

System 

Detectors 
Wavelength 
Scanning  & 
signal 
processing 

Resolution: 

-  Thermal 

-  Spatial 
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Table  10:  Rational  and  comments  -  SAR  systems 


Simulated  Sub 

components  Components 

Required 

fidelity 

Rational  and  comments 

Target 

objects 

Structure 

Materials  & 
texture 

Subsurface 

scattering 

Transmitters 

Background 

Terrain 

Terrain 

objects 

Materials  & 
textures 

Atmosphere 

Platform 

Sensor 

system 

Radar 

system 

parameters 

Interactions 

Actual  fidelity 

(To  be  applied  on  specific  simulated  images). 

General  comments  and  suggestions 


45 


DRAFT 


DRAFT 


Appendix  II  -  Checklist’s  results 

Appendix  II  presents  average  results  and  comments  from  the  investigated  sensor  systems 
and  tasks  of  the  present  validation  study.  The  three  tasks  (photo  interpretation, 
reconnaissance,  target  recognition  and  designation)  produced  similar  result  and  were, 
therefore,  combined.  The  two  sensor  systems  (S  AR  and  FLIR)  by  two  types  of  tasks 
(training  and  mission  rehearsal)  define  the  four  subtasks  presented  below. 

SAR  simulation  for  training 

Number  of  SME:  6 

Type  of  required  validation  criteria:  Variable  (a  generic  criterion  is  irrelevant  because 
real,  rather  than  simulated  images  can  be  used  as  generic  examples.  A  specific  criterion  is 
not  necessary  and  it  is  sufficient  to  present  well-selected  examples  of  representative 
conditions). 

Availability  of  Validation  criteria:  Medium  -  high  (various  real  examples  are  usually 
available,  however,  some  desired  combinations  may  not  be  available). 

Table  11:  Average  required  validity  and  comments  on  a  simulated  SAR,  training 
task. 


Simulated  Sub 

components  Components 

Required 

fidelity 

Rational  and  comments 

Target 

objects 

Structure 

Materials  & 
texture 

Medium  - 
high 

High 

Small  details  cannot  be  seen  in  SAR, 
therefore,  it  is  important  to  simulate  the  size 
and  magnitude  of  scattering,  and  to 
represent  special  cues  (whenever  relevant), 
other  small  details  are  not  crucial. 

Subsurface 

scattering 

low 

Small  nuances  (e.g.  subsurface  scattering) 
are  not  important. 

Transmitters 

Irrelevant 

Most  SAR  systems  are  not  sensitive  to  other 
radar  transmitters. 

Background 

Terrain 

Medium 

Terrain  should  looks  more  or  less  realistic, 
most  specific  details  may  not  be  necessary. 

Terrain 
objects 
Materials  & 
textures 

Very  high  - 
medium 

Terrain  objects  that  participate  in 
interpretation  performance  (e.g.,  defense 
posts)  otherobjects  are  only  as  important  as 
terrain  features. 

Atmosphere 

Very  low 

Has  no  significant  effect  on  SAR 

Platform  & 

Aircraft  and 
flight  path 

Low- 

medium 

The  platform  as  such  is  not  important, 
relevant  effects  should  be  simulated  (e.g., 
deviations  from  flight  path). 

Sensor 

system 

Sensor 

High 

If  training  is  designated  towards  a  specific 
system  the  parameters  of  this  system  must 
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be  simulated. 

Interactions 

Low 

For  practical  purposes,  the  effects  are 
marginal. 

SAR  simulation  for  mission  rehearsal 

Number  of  SME;  2 

Type  of  required  validation  criteria:  Variable  -  specific  (a  specific  criterion  is  desired, 
i.e.,  the  simulated  image  should  be  similar  to  specific,  real  SAR  images,  this  goal, 
however,  is  unrealistic.  A  variable  criterion  may  be  useful  to  some  extent  in  simulating 
relevant  components  and  providing  examples  of  typical  conditions). 

Availability  of  Validation  criteria:  Medium  -  low  (real  images  of  specific  conditions  are 
scarce,  availability  of  images  for  a  variable  criterion  may  be  higher. 

Table  12:  Average  required  validity  and  comments  on  a  simulated  SAR,  mission 
rehearsal  task 


Simulated  Sub 

components  Components 

Required 

fidelity 

Rational  and  comments 

Target 

objects 

Structure, 
Materials  & 
texture 

Medium  - 
high 

Small  details  cannot  be  seen  in  SAR, 
however,  some  targets  may  have  specific 
cues  that  should  be  represented. 

Subsurface 

scattering 

low 

Small  nuances  (e.g.  subsurface  scattering) 
are  not  important. 

Transmitters 

Irrelevant 

Most  SAR  systems  are  not  sensitive  to  other 
radar  transmitters. 

Background 

Terrain 

Terrain 
objects 
Materials  & 
textures 

Medium 

High- 

medium 

Terrain  should  look  realistic,  different  types 
of  areas  must  be  distinguishable.  Highly 
specific  details  may  not  be  necessary. 
Mission-related  terrain  objects  must  be  well 
represented,  other  objects  are  only  as 
important  as  terrain  features.  Hence,  the 
selection  of  objects  for  high-fidelity 
treatment  is  mission  specific. 

Atmosphere 

Very  low 

Has  no  significant  effect  on  SAR 

Platform  & 

Sensor 

system 

Aircraft  and 
flight  path 

Low 

It  must  be  assumed  that  the  operator  is 
familiar  with  the  sensor,  expected  image 
quality,  typical  malfunctions,  etc. 

Sensor 

Very  high 

The  specific  quality  and  parameters  of  the 
relevant  sensor  must  be  represented. 

Interactions 

Low 

Effects  are  marginal  and  practically 
impossible  to  predict. 
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FLIR  simulation  for  training 

Number  of  SME:  6 

Type  of  required  validation  criteria:  Variable  (it  is  important  to  represent  a  representative 
variety  and  a  continuum  of  conditions.  It  is  not  necessary,  however,  to  represent  every 
possible  combination  of  conditions). 

Availability  of  Validation  criteria:  Medium  -  high  (real  examples  are  usually  available, 
however,  some  desired  combinations  may  not  be  available) 


Table  13:  Average  required  validity  and  comments  on  a  simulated  FLIR,  training  task 


Simulated  Sub 

components  Components 

Required 

fidelity 

Rational  and  comments 

Target 

objects 

Structure 
Materials  & 
texture 
Temperature 
&  emissivity 

High  -  very 
high 

Thermal  signatures  depend  on  structure, 
materials,  temperature  and  emissivity. 
Depending  on  the  purpose  of  simulation 
accurate  representations  of  the  real  targets 
must  be  provided. 

Conduction 

Medium 

Heat  conduction  between  parts  of  objects 
creates  the  smooth,  natural  looking 
representation  of  thermal  objects  in  FLIR. 

Time 

functions  & 

changing 

conditions 

High  -  very 
high 

Changes  over  time  and  changes  due  to 
object  condition  (e.g.,  engine  operating  or 
not)  are  very  important.  However,  it  may  not 
be  crucial  to  represent  the  exact  conduct  of 
every  possible  process. 

Background 

Terrain 

Medium 

Terrain  should  look  realistic,  it  must  be 
possible  to  recognize  ground  features  and 
differentiate  between  types  of  terrain. 

Highly  accurate  representation  may  not  be 
necessary. 

Terrain 
objects 
Materials  & 
textures, 
temperature 
& 

emissivity 

High- 

medium 

Mission  related  terrain  objects  must  be  well 
represented,  other  objects  are  only  as 
important  as  terrain  features. 

Thermal 

transfer 

Medium 

Thermal  transfer  induces  the  continues, 
natural-looking  image.  It  may  also  provide 
important  cues  for  detection  and  recognition 
(e.g.,  shading). 
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Heat 

generating 

objects 

Very  high  - 
high 

Heat  generating  objects  are  central  to  many 
military  tasks  and  are  therefore,  highly 
important. 

Reflectance 

Medium  - 
low 

Most  object  do  not  reflect  IR  radiation, 
reflectance  by  water  (sea,  ponds,  etc.) 
should  be  represented. 

Time 

functions. 

High 

Time  related  changes  (e.g.,  hour  of  day  / 
night)  must  be  represented  realistically  but 
do  not  have  to  represent  all  possible 
processes. 

Atmosphere 

Particles 

Medium 

Atmospheric  conditions  have  important 
effects  on  the  image.  For  training  purposes, 
however,  it  may  suffice  to  represent  various 
visibility  levels  without  distinguishing 
between  different  sources  of  interference. 

Platform  & 

Platform 

Medium  - 
high 

It  is  important  that  the  image  provides  an 
accurate  representation  of  real  viewing 
conditions  (i.e.,  angle,  field  of  view, 
movement  parameters,  etc.) 

Sensor 

system 

Detectors, 

Wavelength, 

Scanning  & 

signal 

processing. 

Resolution, 

gain  &  level 

High  -  very 
high 

The  image  must  represent  the  specific 
sensor-system  of  interest  including  its 
functions. 

Interactions 

Medium 

Interactions  between  objects  may  provide 
important  cue,  e.g.,  tracks  on  the  ground, 
shading  effects,  heat  transfer  between 
objects,  etc. 

FLIR  simulation  for  mission  rehearsal 

Number  of  SME:  2 

Type  of  required  validation  criteria:  Specific  (it  is  important  to  represent  the  area  as  it 
will  be  during  the  mission,  otherwise,  the  use  of  simulation  may  not  be  recommended  for 
that  purpose). 

Availability  of  Validation  criteria:  Low  (validation  can  only  be  performed  post-hoc,  i.e., 
after  the  mission). 

Table  14:  Average  required  validity  and  compiled  comments  on  a  simulated  FLIR, 
mission  rehearsal  task 
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Simulated  Sub  Required  Rational  and  comments 

components  Components  fidelity 


Target 

objects 

Structure 
Materials  & 
texture 
Temperature 
&  emissivity 

High  -  very 
high 

Thermal  signatures  depend  on  structure, 
materials,  temperature  and  emissivity. 
Therefore,  target  objects  should  be 
represented  as  accurately  as  possible. 

Conduction 

Medium 

Heat  conduction  between  parts  of  objects 
creates  the  smooth,  natural  looking 
representation  of  thermal  objects  in  FLIR. 

Time 

functions  & 

changing 

conditions 

High  -  very 

The  conditions  during  the  mission  should  be 
anticipated,  including  time  effects  and  other 
changing  effects. 

Background 

Terrain 

Medium 

Given  that  the  mission  is  defined,  it  may  be 
possible  to  invest  most  of  the  efforts  in 
relevant  rather  than  in  all  terrain  features. 

Terrain 
objects 
Materials  & 
textures, 
temperature 
& 

emissivity 

High- 

medium 

Given  that  the  mission  is  defined,  it  may  be 
possible  to  invest  most  of  the  efforts  in 
important  terrain  objects  rather  than  in  all 
objects. 

Thermal 

transfer 

Medium 

Thermal  transfer  generates  continues, 
natural-looking  image.  It  may  also  contain 
important  cues  for  detection  and  recognition 
(e.g.,  shading). 

Heat 

generating 

objects 

Very  high 

Heat  generating  objects  are  highly  important 
in  most  military  tasks.  It  may  be  required  to 
distinguish  between  very  similar  objects 
(e.g.,  real  targets  and  decoys) 

Reflectance 

Medium  - 
low 

May  be  important  if  the  mission  takes  place 
near  open  water  bodies  (sea,  lake,  etc.). 

Time 

functions. 

High 

Time  related  changes  should  be  anticipated 
in  order  to  represent  a  realistic  image  of 
terrain  and  terrain  objects  during  the 
mission. 

Atmosphere 

Particles 

High- 

Visibility  conditions  during  the  mission 
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medium 

should  be  anticipated.  Alternatively,  the 
operator  may  prepare  for  a  variety  of 
conditions. 

Platform  & 

Platform 

High 

The  image  provides  an  accurate 
representation  of  the  relevant  features  of  the 
platform  (e.g.,  angle,  field  of  view, 
movement  parameters,  etc.) 

Sensor 

system 

Detectors, 

Wavelength, 

Scanning  & 

signal 

processing. 

Resolution, 

gain  &  level 

High  -  very 
high 

The  image  must  represent  the  specific 
sensor-system  of  interest  including  its 
functions. 

Interactions 

Medium 

Interactions  between  targets  and  terrain  may 
provide  important  cues,  e.g.,  tracks  on  the 
ground,  shading  effects,  etc. 
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